Managing Failure, Avoiding Failure: Learning From Our Mistakes.

Posted: April 12, 2012 | Author: | Filed under: Education | Tags: , , , , , , | 3 Comments

One thing that really identifies the best of my students is their willingness to try – they’ll take what they’ve learnt and throw themselves into unknown situations and see what happens.

Of course, to be willing to do this you either know what it is to fail and have learned not to be scared of it – or you’ve never failed before. I don’t think anyone’s defined by their first reaction to failure. Or their second…or their third… of course, all of this assuming that they were genuinely trying and preparing and (with any luck) learning and improving.

Let’s be honest – failure sucks. It’s more than the absence of success, because it’s the fact that you actually tried to succeed. Right now, I am failing in my attempt to be the first man to walk on Mars, but I’m really not all that upset because I haven’t spent all that much time on it. It’s not really a goal for me so I’m not invested in it. I’ve got some stuff that I am invested in and, if I fail at that, then that is going to suck but I’ll try to learn as much as I can from it.

I’m lucky, I’ve failed a lot but at times and in ways that haven’t wiped me out. It took me a long time to learn how to manage failure and an even longer time to work out how lucky I was that the nature of failure I experienced was so benign. So, reading this, keep in mind that I’m talking about academic marks, not ‘making a bad decision, killing two people and ending up in jail for the rest of my life’. It’s important to keep this in perspective, sometimes.

A lot of my students have a pretty good run on their way to my class. Top of their school classes, relatively comfortable background, low-crime environment, nice weather. Some of my students will encounter their first major failure while they’re with me. For some students, this is going to be shattering. This isn’t a matter of “it’s a bad report card”, it could be ‘your degree will take another 12 months.” The worst case is “you can’t study with us anymore” – but that’s a very serious proposition that’s only reached after years of underperformance.

I understand this because my first major failures happened when I hit Uni. Suddenly I had to study, I had to do things, I couldn’t get by. It took me a while to get my head around it but, in doing so, I really learnt the difference between those lecturers who could help me and those who couldn’t. I went on to succeed but I also went on to keep trying, even when I was reaching higher and my risk of failure became more severe. Here’s a sort list of the things that I try to pass on but there’s a lot more I could say on this:

  1. Don’t just say “I should have done more” identify what you should have done.
    “I will work harder” is meaningless – you’ll work harder at what? “I will start my assignments 1 week earlier” is a clear statement of what you need to do but is secretly composed of so many other assumptions. “I will review my assignments on the day I get them and add them to my assignment plan (see point 2)” starts the process but, well, it’s a rabbit hole that varies from person to person in its depth. Don’t know what you should have done? Find someone you trust, ask them and then listen to the answer (see point 4). A clear statement of what you need to do will help you to focus and, with work and luck, achieve a better result next time. It also gives you something to direct your energies towards – work out those negative vibes!
  2. Not all tasks are equal in importance, opportunity or consequence. 
    Want a degree? Then, at times, your coursework is more important that anything else. Want good marks? Then you’re going to have to move other tasks aside to make time to study for the exam. To do this, you’re going to need to plan and get stuff sorted out far enough in advance that you’re not wasting all of your spare time (that you’re trying to make) running around trying to free up the spare time (that you now don’t have). There are many platitudes around planning, and its importance, but planning properly is generally the first requirement for success. Practice it, get good at it – it almost always makes things better.
  3. Sleep isn’t optional.
    Caffeine is amazing but sleep is the real thing. Very few of us can get by with little sleep and high-level mental activity generally needs the right amount of sleep, on a regular basis. Not sleeping for ages and taking caffeine tablets just makes you jittery and stupid. Believe me, I know. Sleep makes you less grumpy, more mentally agile, happier and lets you see your real problems in their proper size. Why, I’m sleeping as I type this!
  4. Find the right person to ask about how you can improve.
    Everyone has an opinion. Some people have the right opinion. You have to work out the right people to speak to, who have the knowledge you need and can present it in a way that you can then absorb. This combination of message+medium will vary from person to person. Some people who have never failed have great empathy and can help you anyway. Some people, well, let’s just say that some people will look at you like you’re some kind of alien and how on Earth could you fail something but I guess I could suggest that you work harder because it’s important – seriously? Forget them. Mentors, role models and guides are real people with real lessons to share – find someone authentic. Not everyone has to have feet of clay – but a little dust on their wings won’t go amiss.
  5. Be firm but fair with yourself.
    Yes, you failed something. Did you actually try? No? Why not? Don’t say “It’s ok because I didn’t try” – if you don’t want to try, don’t play. Do something else that you really care about. Being magnificent somewhere else is always a good option! If you did try – what happened? (See point 1) Yes, you failed – don’t beat yourself but, at the same time, don’t forget it straight away either. Remember what it was that tripped you up and try to stop it from happening again. If you find yourself waking up at night, sweating, thinking about it: you’re not being fair to yourself. Obsession isn’t fair to yourself. At the same time, saying “Meh” and going round in unproductive cycles isn’t that fair either – unless that’s what makes you happy and you have the money to burn. In which case, well, have fun but try not to drag anyone else down with you because most people in your course want to pass and I’d rather that they didn’t pick up those habits from you. Nothing personal but passing students is what I try to do. 🙂

So much more I could write here but I don’t want to go on too much. Hope that this is interesting for you.

A Dangerous Precedent: Am I Expecting Too Much of My Students?

Posted: April 11, 2012 | Author: | Filed under: Education | Tags: , , , , , , , | 2 Comments

Anyone with a pulse is aware that there is a lot of discussion at the moment in some important areas of Science. If we scratch the surface of the climate and vaccination debates, we find a roiling frenzy of claim and counter-claim – facts, fallacies and fury all locked in a seething ball. We appear to have reached a point where there is little point in trying to hold a discussion because we have reached a point of dogmatic separation of the parties – where no discussion can bridge the divide. This is the dangerous precedent I’m worried about – not that we have contentious issues, but that we have contentious issues where we build a divide that cannot be bridged by reasonable people with similar backgrounds and training. This is a sad state of affairs, given the degree to which we all observe the same universe.

I don’t teach politics in the classroom and I try not to let my own politics show but I do feel free to discuss good science with my students. Good science is built on good science and, ultimately, begets more good science. Regrettably, a lot of external interest has crept in and it’s easy to see places where good science has been led astray, or published too early, or taken out of context. It’s also easy to see where bad science has crept in under the rug disguised as good science. Sometimes, bad science is just labelled good science and we’re supposed to accept it.

I’m worried that doubt is seen as weakness, when questioning is one of the fundamental starting points for science. I’m worried that a glib (and questionable) certainty is preferred to a complex and multi-valued possibility, even where the latter is correct. I’m worried that reassessment of a theory in light of new evidence is seen as a retrograde step.

I have always said that I expect a lot of my students and that’s true. I tell my research students that will work hard when they’re with me, and that I expect a lot, but that I will work just as hard and that I will try to help them achieve great things. But, along with this, I expect them to be good scientists. I expect them to read a lot across the field and at least be able to make a stab at separating good, replicable results from cherry-picking and interest-influenced studies. That’s really hard, of course, especially when you read things like 47 of the most significant 53 cancer studies can’t be replicated. We can, of course, raise standards to try and address this but, if we’re talking about this in 2012, it’s more than a little embarrassing for the scientific community.

What I try to get across to my students is that, in case of pressure, I expect them to be ethical. I try to convey that a genuine poor (or null) submission is preferable to an excellent piece of plagiarised work, while tracking and encouraging them to try and stay out of that falsely dichotomous zone. But, my goodness, look at the world and look at some of the things we’ve done in the name of Science. Let’s look at some of those in the 20th century with something approaching (semi)informed consent. The Tuskegee Syphilis Experiment. Milgram’s experiment. The Stanford Prison Experiment. I discuss all of these with my students and a number of them think I’m making it up. Until they go looking.

Now, as well as unethical behaviour to contend with, we have divisive behaviour – people trying to split the community for their own purposes. We always had it, of course, but the ease of self-publishing and the speed with which information can be delivered means that it takes days to spread information that used to percolate through doubt filters and peer review. Bad science can often travel faster than good science because it bypasses the peer review process – which has been unfairly portrayed in certain circles as an impediment to innovation or a tool of ‘Big Science’. The appeal to authority is always dangerous, because there is no guarantee that peer review is flawless, but as we have seen with the recent “Faster than the speed of light/ oh, wait, no it’s not” the more appropriately trained eyes you have on your work, the more chance we have of picking up mistakes.

So I expect my students to be well-read, selective, ethical, inclusive and open to constructive criticism as they work towards good or great things.

I still believe that there is a strong and like-minded community out there for them to join – but some days, reading the news, that’s harder to believe than others.

The Student of 2040

Posted: April 10, 2012 | Author: | Filed under: Education | Tags: , , , , , , | 1 Comment

On occasion, I wonder about where students and teachers will be towards the end of my teaching career. Let us be optimistic and say that I’ll still be teaching in 2040 – what will my students look like? (I’m not sure that teaching at that age is considered optimistic but bear with me!)

A person with electrodes on their head.

Right now we’re having to adapt to students who can sit in lectures and, easily and without any effort, look things up on wireless or 3G connections to the Internet – searching taking the role of remembering. Of course, this isn’t (and shouldn’t be) a problem because we’re far more than recitation and memorisation factories. There are many things that Wikipedia can’t do. (Contrary to what at least a few of my students believe!)

But what of the future? What of implanted connections that can map entire processes and skills into the brain? What of video overlays that are invisibly laid over the eye? Very little of what we use today to drive thinking, retrieval, model formation and testing will survive this kind of access.

It would be tempting to think that constant access to the data caucus would remove the need for education but, of course, it only gives you answers to questions that have already been asked and answers that have already been given – a lot of what we do is designed to encourage students to ask questions in new areas and find new answers, including questioning old ones. Much like the smooth page of Wikipedia gives the illusion of a singularity of intent over a sea of chaos and argument, the presence of many answers gives the illusion of no un-answered questions. The constant integration of information into the brain will no more remove the need for education than a library, or the Internet, has already done. In fact, it allows us to focus more on the important matters because it’s easier to see what it is that we actually need to do.

And so, we come back to the fundamentals of our profession – giving students a reason to listen to us, something valuable when they do listen and a strong connection between teaching and the professional world. If I am still teaching by the time I’m in my 70s then I can only hope that I’ve worked out how to do this.

Sitting Alone in a Crowd Just Like You: The Isolated Student

Posted: April 9, 2012 | Author: | Filed under: Education | Tags: , , , , , , , | 6 Comments

I’ve posted a number of things about the importance of training people to use all the techniques and technologies that we have – which applies to us teaching as much as it ever applies to the student. The best way to training and education to happen is, unsurprisingly, provide the best environment in which this can happen – resources, buildings, rooms and, above all, community. One of the big benefits of the traditional lecture is that we get the whole student community for a course into the one room, several times a week. What isn’t guaranteed is that we will actually get a proper community forming – we may just have a big room full of people, none talking to each other, no-one interacting. At that stage, of course, we have the traditional lecture at it’s potential worst.

I’m writing a series of short stories at the moment around the theme of isolation and there are many excellent examples in literature of the notion of being alone while in the company of others. Whether it’s the isolation of the protagonist in Bashevis Singer’s The Slave, separated from the community by his religion and status, or the isolation of the reader from the world and characters from Philip K. Dick, a wonderful example of umwelt, we see the same notion over and over again – the presence of other people, even like-minded people, does not guarantee any protection from isolation because of issues with perception, background and expectations.

There are many reasons that a student can feel that they can’t reach out to their community and handling this is one of the big issues that we have in dealing with the transition into a new course of studies. We spend a lot of time with (very useful) mentoring schemes, networking among our students with supporting BBQs and meet-ups – but the things we actually do in lectures are also very helpful. If you have a group activity, or a talk-to-your-neighbour, exercise in the first week, you can be setting up small communities and networks that will survive for the rest of the student’s career. If the lecture space is also a community space, a meeting place and something that people look forward to, then we start to work on the many possible issues that can lead to isolation.

If someone isn’t making progress then, rather than focus on what they’re not doing and giving them a relatively tired lecture on keeping up effort and putting in the hours, it can sometimes help to ask who they’re working with, whether they have any friends in the course and, if not, direct the student to your support system. For students who have more serious problems, at our University we have a Transitions Service who provide a shopfront to the other services we have, ranging from friendly chats over tea to introductions to our counselling service. This is, however, a little bit of overkill in most cases – a chat, some activity in lectures and a BBQ or two can address at least some of the problems.

Educational Software Systems: What are our requirements?

Posted: April 8, 2012 | Author: | Filed under: Education | Tags: , , , , , , , , , | 1 Comment

My recent evaluation of strategic IT issues in my faculty brought one thing very clearly to my attention. If we assume that the content that I (or a student or any other academic) creates should continue to be available to me, unless I assign my rights to someone else, then we have a problem if the storage mechanism used is closed and fee-based. If it’s closed (proprietary formats possibly with deliberate obfuscation or encryption, or remote storage with access controls) then I can’t easily get it out of the system unless the software provider lets me. If this whole arrangement (licensing or access) is based on a fee, the the worst possible situation is that, in order to access old materials/data/whatever, I have to continue to pay largish sums on money to keep using something that I created. So that got me thinking – what else do I expect (naively or not) of the systems that we use for education? I’m talking mostly about Learning Management Systems (LMS) here because that’s my current focus. Here are some cut down versions of my current thoughts.

  1. Modular: Not all schools, even in the same college, are the same. Some are big, some are small. Some need essays checked for plagiarism when submitted. Some just need a place to drop the assignments. Some need interactive quizzes! Some just need web pages. If we have a system that’s built out of modules then I can get the modules that I need and (if money is involved) match investment to requirement. Even in the open source community, there are issues of performance, management burden and complexity in the modules that you choose so customisation here is useful. Modularity also isolates the impact of faults. Well-designed modules are like water-tight doors on a ship – one module failing doesn’t sink the ship.
  2. Extensible: There are always going to be requirements particular to your school or college. If your system can be extended then you can adjust the system to meet your specific needs. If not, then you have to work around it. (You could also call this mutable but I prefer the improvement implicit in extensible). Now I’m not saying who should be doing these changes because that’s a whole other argument – the fact that it can be done is what’s important here.
  3. Open Storage: Whatever I create, I should be able to get to, export, import it back from other systems and hang onto – especially if we migrate to a different system and shut the old one down. I’m a great believer in keeping formats open and then, if your product is excellent, I’ll happily use it. If, at the start of our relationship, you say “Well, you give me your data and you’re going to have to pay us if you want to get it out, and pay us every year you want access to it.” then you are pretty much going to have to be the only game in town because I have no power or control in that relationship – and I’m the one who should be in control here. (Most people are getting really good about this now, which is good to see, but there are far too many examples of software where the same producer couldn’t maintain compatibility across two versions.)
  4. Efficient: Using these kinds of systems should save you time. Something I’ve created should be able to be re-used easily, live or die as quickly as I want and survive between upgrades. There should be a definite advantage to doing this – if not, why are we using this system?
  5. Robust: It should be strong. Students attack in waves and I have beautiful graphs to show that the day before an assignment is due it’s “STOP, Hammer time”, and weak or delicate systems will fail under this kind of load. Any system designer who has assumed an average and hardened the system up to 70% concurrent usage had better step back and add about 40% to that number to count for multiple accesses, dangling connections, staff use… Even more importantly, if I make a change to something (add an assignment, undertake an assignment, change a mark) it needs to stay CHANGED. Finally, I need to be able to undo things when and if they go wrong. Because things go wrong.
  6. Invisible: Ultimately, I shouldn’t notice the system, and nor should my students. I use it to create things but I focus on creation, not your system. My students use it to access things and perform actions but they should never notice the system itself. If you look out a window, you should never notice cracked panes, dirty glass, rotting wood or the accidental sandwiching of a bird between the double-glazed panes.

I realise that there is a lot of debate on Open versus Proprietary systems in the community and I have no wish to open that can of worms – for myself, I use a Mac (mostly on the FreeBSD side) and do my analysis work in R, not S, but then I use Illustrator, not the Gimp, for final touch-ups. I’m the poster boy for using stuff that works so I have no wish to force people to choose open or closed, or ‘whatever is in between’. But I’d be really interested to see what else belongs (or what doesn’t belong) on this list – what do you think?

How the Internet Works: For You and On You

Posted: April 7, 2012 | Author: | Filed under: Education | Tags: , , , , , , | 1 Comment

I would link to a recent April 1st post on Randall Munroe’s XKCD (it’s http://www.xkcd.com/1037/ if you need to know) but I can’t because I’m not sure what you’ll see when you get there and some of the possible images are not safe for work. The theme of his post is umwelt, the notion that your perception of your own world is highly personal, influenced by your previous experiences and the lens that you have on your environment.

This is the image I see now – using my default browser. It is, very much, what you make of it. (I attribute the image to XKCD, but I can’t link to it for obvious reasons.)

A well? The Ring? The Sun? Your perception of what this mean depends on your senses, your experiences and your world to date.

Some people saw images of unlikely earthquakes. Some saw images of groups being stalked by velociraptors – but only when they elongated their screens enough to see everything on the page. Serving military saw a supportive post that encouraged them to keep an eye on the missiles. Panels adjusted based on who you were, which browser you used, where you were, even what size screen you used.

How did this work? Your browser provides a great deal of information as to what it is, to allow sites to adjust for browser variation. At the same time, the Internet Protocol (IP) address of your computer, the address that other computers use to get to it, gives away your location and your organisation. That’s why students at MIT got MIT jokes and people in Israel got an XKCD comic in Hebrew.

I can’t think of a better way to introduce students to the idea that gazing into the abyss also involves the abyss gazing into you. There are two great summary pages here (Reddit) and here (Google document).

This is the Internet and it’s ubiquitous. This is a great way to discuss the amount of information that is being sent back by every browser, from every platform. The amount of information that can be obtained by underlying systems that most people are unaware of.

For me, what I found most amusing is that a friend sent the link of the image to me saying “Hey, Voight Kampf test” (from Blade Runner) and I saw a picture of something completely different and thought my friend was mad. That’s something else to talk about – what it would mean if information itself mutated depended on who you were and where you were.

Putting the pieces together: Constructive Feedback

Posted: April 6, 2012 | Author: | Filed under: Education | Tags: , , , , , , , | Leave a comment

Over the last couple of days, I’ve introduced a scenario that examines the commonly held belief that negative reinforcement can have more benefit than positive reinforcement. I’ve then shown you, via card tricks, why the way that you construct your experiment (or the way that you think about your world) means that this becomes a self-fulfilling prophecy.

Now let’s talk about making use of this. Remember this diagram?

This shows what happens if, when you occupy a defined good or bad state, you then redefine your world so that you lose all notion of degree and construct a false dichotomy through partitioning. We have to think about the fundamental question here – what do we actually want to achieve when we categorise activities as good or bad?

Realistically, most highly praiseworthy events are unusual – the A+. Similarly, the F should be less represented across our students than the P and the “I’ve done nothing, I don’t care, give me 0” should be as unlikely as the A+. But, instead of looking at outcomes (marks), let’s look at behaviours. Now I’m going to construct my feedback to students to address the behaviour or approach that they employed to achieve a certain outcome. This allows me to separate the B+ of a gifted student who didn’t really try from the B+ of a student who struggled and strived to achieve it. It also allows me to separate the F of someone who didn’t show up to class from the F of someone who didn’t understand the material because their study habits aren’t working. Finally, this allows me to separate the A+ of achievement from the A+ of plagiarism or cheating.

Under stress, I expect people to drop back down in their ability to reason and carry out higher-level thought. We could refer to Maslow or read any book on history. We could invoke Hick’s Law – that time taken to make decisions increases as function of the number of choices, and factor in pilot thresholds, decision making fatigue and all of that. Or we could remember what it was like to be undergraduates. 🙂

Let me break down student behaviours into unacceptable, acceptable and desirable behaviours, and allocate the same nomenclature that I did for cards. So the unacceptable are now the 2,3,4, the acceptable and 5-10, and the desirable are J-K. (The numbers don’t matter, I’m after the concept.) When I construct feedback for my students, I want to encourage certain behaviours and discourage others. Ideally, I want to even develop more of certain behaviours (putting more Kings in the deck, effectively). But what happens if I don’t discourage the unacceptable behaviours?

Under stress, under reduced cognitive function and reduced ability to make decisions, those higher level desirable behaviours are most likely going to suffer. Let us model the student under stress as a stacked deck, where the Q and K have been removed. Not only do we have less possibility for praise but we have an increased probability of the student making the wrong decision and using a 2,3,4 behaviour. It doesn’t matter that we’ve developed all of these great new behaviours (added Kings), if under stress they all get thrown out anyway.

So we also have to look at removing or reducing the likelihood of the unacceptable behaviours. A student who has no unacceptable behaviour left in their deck will, regardless of the pressure, always behave in an acceptable manner. If you can also add desirable behaviours then they have an increase chance of doing well. This is where targeted feedback comes in at the behavioural level. I don’t need to worry about acceptable behaviours and I should be very careful in my feedback to recognise that there is an asymmetric situation where any improvement from unacceptable is great but a decline from desirable to acceptable may be nowhere near as catastrophic. Moderation, thought and student knowledge are all very useful concepts here. But we can also involve the student in useful and scaleable ways.

Raising student awareness of how their own process works is important. We use reflective exercises midway through our end-of-first year programming course to develop student process awareness: what worked, what didn’t work, how can you improve. This gives students a framework for self-assessment that allows them to find their own 2s and their own Kings. By getting the students to do it, with feedback from us, they develop it in their own vocabulary and it makes it ideal for sharing with other students at a similar level. In fact, as the final exercise, we ask students to summarise their growth in their process awareness during the course and ask them for advice that they would give to other students about to start. Here are three examples:

Write code. Go beyond your practicals, explore the language, try things out for yourself. The more you code, the better you’ll get, so take every opportunity you can. 

If I could give advice to a future … student, I would tell them to test their code as often as possible, while writing it. Make sure that each stage of your code is flawless before going onto the next stage. 

If I could give one piece of advice to students …, it would be to start the coding early and to keep at it. Sometimes you can have a problem that you just can’t work out how to fix but if you start early you leave yourself time to be able to walk away and come back later. A lot of the time when you come back you will see the problem very quickly and wonder why you couldn’t see it before. 

There are a lot of Kings in the above advice – in fact, let’s promote them and call them Aces, because these are behaviours that, with practice, will become habits and they’ll give you a K at the same time as removing a 2. “Practise your coding skills” “Test your code thoroughly” “Start early” Students also framed their lessons in terms of what not to do. As one student reflected, he knew that it would take him four days to write the code, so he was wondering why he only started two days out – you could read the puzzlement in the words as he wrote them. He’d identified the problem and, by semester’s end, had removed that 2.

I’m not saying anything amazing or controversial here, but I hope that this has framed a common situation in a way that is useful to you. And you got to shout at cards, too. Bonus!

Card Shouting: Teaching Probability and Useful Feedback Techniques by… Yelling at Cards

Posted: April 5, 2012 | Author: | Filed under: Education | Tags: , , , , , , , , , | 2 Comments

I introduced a teaching tool I call ‘Card Shouting’ into some of my lectures a while ago – usually those dealing with Puzzle Based Learning or thinking techniques. What does Card Shouting do? It demonstrates probability, shows why experimental construction is important and engages your class for the rest of the lecture. So what is Card Shouting?

(As a note, I’ve been a bit rushed today and I may not have put this the best way that I could. I apologise for that in advance and look forward to constructive suggestions for clarity.)

Take a standard deck of cards, shuffle it and put it face down somewhere where the class can see it. (You either need very big cards or a document camera so that everyone can see this in a large class. In a small class a standard deck is fine.)

Then explain the rules, with a little patter. Here’s an example of what I say: “I’m sick and tired of losing at Poker! I’ve decided that I want to train a pack of cards to give me the cards I want. Which ones do I want? High cards, lots of them! What don’t I want? Low cards – I don’t want any really low cards.”

Pause to leaf through the deck and pull out a Ace, King and Queen. Also grab a 2,3 and 4. Any suit is fine. Hold up the Ace, King and Queen.

“Ok, these guys are my favourites. What I’m going to do, after this, is shuffle everything back in and draw a card from the top of the deck and flip it up so you can see it. If it’s one of these (A, K, Q), I want to hear positive things – I want ‘yeahs!’ and applause and good stuff. But if it’s one of these (hold up 2,3 and 4) I want boos and hisses and something negative. If it’s not either – well, we don’t do anything.” (Big tip: put a list of the accepted positive and negative phrases on the board. It’s much easier that having to lecture people on not swearing)

“Let’s find out what works best: praise or yelling- but we’re going to have measure our outcomes. I want to know if my training is working! If I draw a bad card and abuse it, I want to see improvement – I don’t want to see one of the bad cards on the next draw. If I draw a good card, and praise it, I want the deck to give me another good card – if it’s lower than a J, I’ve wasted my praise!” Then work through with the class how you can measure positive and negative reaction from the cards – we’ve already set up the basis for this in yesterday’s post, so here’s that diagram again.

In this case, we record a tick in the top left hand box if we get a good card and, after praising it, we get another good card. We put a tick in top right hand box if we draw a bad card, abuse the deck, and then get anything that is NOT a bad card. Similarly for the bottom row, if we praise it and the next card is not good, we put a mark in the bottom left hand box and, finally, if we abuse the card and the next card is still bad, we record a mark in the bottom right hand box.

Now you shuffle and start drawing cards, getting the reactions and then, based on the card after the reaction, record improvement or decline in the face of praise and abuse. If you draw something that’s neither good NOR bad to start with – you can ignore it. We only care about the situation when we have drawn a good or a bad card and what the immediate next card is.

The pattern should emerge relatively quickly. Your card shouting is working! If you get a card in the bad zone, you shout at it, and it’s 3 times as likely to improve. But wait, praise doesn’t do anything for these cards! You can praise a card and, again, roughly 3 times as often – it gets worse and drops out of the praise zone!

Not only have we trained the deck, we’ve shown that abuse is a far more powerful training tool!

Wait.

What? Why is this working? I discussed this in outline yesterday when I pointed out it’s a commonly held belief that negative reinforcement is more likely to cause beneficial outcome based on observation – but I didn’t say why.

Well, let’s think about the problem. First of all, to make the maths easy, I’m going to pull the Ace out of the deck. Sorry, Ace. Now I have 12 cards, 2 to K.

Let me redefine my good cards as J, Q, K. I’ve already defined my bad cards as 2, 3, 4. There are 12 different card values overall. Let me make the problem even simpler. If we encounter a good card, and praise it, what are the chances that I get another good card? (Let’s assume that we have put together a million shuffled decks, so the removal of one card doesn’t alter the probabilities much.)

Well, there are 3 good card values, out of 12, and 9 card values that aren’t good – so that’s 9/12 (75%) chance that I will get a not good card after a good card. What about if I only have 12 cards? Well, if I draw a K, I now only have the Q and the J left in my good card set, compared to 9 other cards. So I have a 9/11 chance of getting a worse card! 75% chance of the next card being worse is actually the best situation.

You can probably see where this is going. I have a similar chance of showing improvement after I’ve abused the deck for giving me a bad card.

Shouting at the cards had no effect on the situation at all – success and failure are defined so that the chances of them occurring serially (success following success, failure following failure) are unlikely but, more subtly, we extend the definition of bad when we have a good card – and vice versa! If we draw any card from 5 to 10 normally, we don’t care about it, but if we want to stay in the good zone, then these 6 cards get added to the bad cards temporarily – we have effectively redefined our standards and made it harder to get the good result. Here’s a diagram, with colour for emphasis, to show the usually neutral outcomes are grouped with the extreme.. No wonder we get the results we do! The rules that we use have determined what our outcome MUST look like.

So, that’s Card Shouting. It seems a bit silly but it allows you to start talking about measurement and illustrate that the way you define things is important. It introduces probability in terms of counting. (You can also ask lots of other questions like “What would happen if good was 8-K and bad was 2-7?”)

My final post on this, tomorrow, is going to tie this all together to talk about giving students feedback, and why we have to move beyond simple definitions of positive and negative, to focus on adding constructive behaviours and reducing destructive behaviours. We’re going to talk about adding some Kings and removing some deuces. See you then.

It’s Feedback Week! Yelling at Pilots is Good for Them!

Posted: April 4, 2012 | Author: | Filed under: Education | Tags: , , , , , , , , , | 1 Comment

The next few posts will deal with effective feedback and the impact of positive and negative reinforcement. I’ll start off with a story that I often tell in lectures.

There’s a well established fact among pilot instructors, in the Air Force, that negative reinforcement works better than positive reinforcement. Combat pilots  are very highly trained, go through an extremely rigorous selection procedure and have to maintain a very high level of personal discipline and fitness. The difference between success and failure, life and death effectively, can come down to one decision. During training, these pilots are put under constant stress, to try and prepare them for the real situation.

A pilot

This man redefines the phrase
"Highway to the Danger Zone".

The instructors have observed that these pilots respond better to being yelled at than being congratulated. What’s their evidence? Well, if a pilot does something really well, and is praised, chances are his or her performance will get worse. Whereas, if the pilot does something bad, or dangerous, and you yell at him or her, his or her performance will improve. Well, that’s a pretty simple framework. Let’s draw it as a diagram.

Now, of course, to see the impact of praise or abuse, you have to record what you did (the praise or abuse) and then what happened (improvement or decline). So we need to draw up some boxes. Ticks mean praise, crosses mean abuse. Up arrow mean improvement, down arrow means decline. A tick in the box that you reach by selecting the column that belongs to Praise (tick) and the row that belongs to Improvement (up arrow) means that the pilot improved after praise. Let’s look at a picture.

So here we can see that, much as an instructor would expect, when I’m nice to you, you get worse more often that you get better. You get better once when praised, compared to getting worse three times. But, wow, when I yelled at you, you get better far more often than you got worse!

There is, of course, a trick here. Yes, it appears that shouting works better than praising but, without giving the game away in the comments (or Googling), do you know why? (The data is a reasonable approximation of the real situation, so there are no hidden arrows or ticks anywhere. 🙂 )

Now, true confession, the picture above is not actually of pilot training – I don’t have an F18 on my desk – but it will be a reasonable approximation of the situation, although it comes from a different source. The graph above comes from a game called “Card Shouting” and I’ll tell you more about that in tomorrow’s post.

Improving, Holding Steady and Going Downhill – Giving Students Useful Feedback

Posted: April 3, 2012 | Author: | Filed under: Education | Tags: , , , , , , , | Leave a comment

I’ve written before about the slightly fuzzy nature of marks but that, overall, we can roughly class marks into ‘failing’, ‘doing okay’ and ‘doing really well’, One thing that I think is really useful is giving students an indication of how they are going in terms of getting better, staying where they are or falling behind.

This is hard throughout courses unless we’ve done some important things, and we’ve also committed to some things across an entire set of courses, like a degree. I’ve covered some of these before but this is a lot more focused.

  1. We’ve tied the assessment firmly into the course so that success in one aspect is a reasonable indicator of continued success.
  2. Students can get early indication when they’re not getting it – whether it’s quick quizzes, feedback on assignments or activities in lectures. Early warning signs are always there.
  3. WE follow up on the early warning signs as well to warn students of what’s going on.

It’s the whole instrumentation, measurement and action routine that I’ve been banging on about for three months now. But let’s put it into a tighter framework, in some senses, with a looser measurement system. We’re only worried about improvement, stability or decline. But is it ever that simple?

I honestly don’t really care if students get 87 or 90 in many ways – High Distinction is High Distinction. If a student gets a series of marks as 86, 91, 88, 90 they’re holding steady. I certainly don’t want them banging on my door, lamenting their decline, if their next mark is an 85 – this is stable and it’s good. But what about this sequence: 50, 52, 57, 53? This is a much riskier proposition, of course, because it’s so much closer to the fail line. Being under 55 is wandering into the zone where one bad mark or missed question could fail you. You don’t want to be stable here.

So, even with a simple three-way framework – it’s pretty obvious that stability is relative. What we really have is different zones where only some of these activities are valid, which should come as no surprise to anyone. 🙂

Below 60, stability doesn’t really cut it and decline is completely unacceptable. Below 60, you really want to be above 60. Yes, 50-60 is a pass but it’s also an indicator that you’re just scraping by – an unlucky day could cost you 6 months of work. Above 60, up to say 75? Stability is ok but we should really be aiming for improvement – if the student can. Above 75? Well, some people will never get much beyond that and all their striving will result in a hard-earned stability that may still taste a little bitter at times. This is where our knowledge of the student comes in.

Not knowing a student’s ability means that you risk telling them to improve when there’s nothing else left and they’ve given all they can. So let me throw out my classification framework and replace it with two questions for the student:

How do you feel about your mark?

Do you think you could have done better?

Balancing that with your knowledge of the student, and guiding them through the thinking process, will give them a better idea of what they can and can’t do. Did they really struggle to get that 66? No? Well, they could have worked harder and maybe got a better mark. That 75 nearly killed them and they really put their all into it? Well that’s one heck of a fine mark.

We all know that very few people know themselves and that’s why I like to try and help them understand why they’re doing what they’re doing and, maybe, once in a while, help them to either accept the fruits of their labours, or to strive that little bit more, if they still have something left to strive with and think it’s worthwhile.

← Older Entries
Newer Entries →