Sources of Knowledge: Stickiness and the Chasm Between Theory and Practice.

Posted: October 21, 2012 | Author: | Filed under: Education | Tags: , , , , , , , , , , , , , , , , | Leave a comment

Like all sources, it helps to know the origin and the purity.

My head is still full of my current crop of research papers and, while I can’t go into details, I can discuss something that I’m noticing more and more as I read into the area of Computer Science Education. Firstly, how much I have left to learn and, secondly, how difficult it is sometimes to track down ideas and establish novelty, provenance and worth. I read Mark Guzdial’s blog a lot because Mark has spent a lot of time being very clever in this area (Sorry, Mark, it’s true) but he is also an excellent connecter of the reader to good sources of information, as well as reminding us when something pops up that is effectively a rehash of an old idea. This level of knowledge and ability to discuss ideas is handy when we keep seeing some of the same old ideas pop up, from one source or another, over time. I’ve spoken before about how the development of the mass-accessible library didn’t end the importance of the University or school, and Mark makes a similar note in a recent post on MOOCs when he points us to an article on mail delivery lessons from a hundred years before and how this didn’t lead to the dissolution of the education system. Face-to-face continues to be important, as do bricks and mortar, so while the MOOC is a fascinating new tool and methodology with great promise, the predicted demise of the school and college may (once again) turn out to be premature.

If you’ve read Malcolm Gladwell’s “The Tipping Point”, you’ll be familiar with the notion that ideas need to have certain characteristics, and certain human agents, before they become truly persuasive and widely adopted. If you’ve read Dawkin’s “Selfish Gene” (published over a decade before) then you’ll understand that Gladwell’s book would be stronger if it recognised a debt to Dawkins’ coining of the term meme, for self-replicating beliefs and behaviours. Gladwell’s book, as a source, is a fairly unscientific restatement of some existing ideas with a useful narrative structure, despite depending on some now questionable case studies. In many ways, it is an example of itself because Gladwell turned existing published information into a form where, with his own additions, he has identified a useful way to discuss certain systems of behaviour. Better still, people do (still) read it.

(A quick use of Google Trends shows me that people search for “The Tipping Point” roughly twice as much as “The Selfish Gene” but for “Richard Dawkins” twice as much as “Malcolm Gladwell”. Given Dawkins’ very high profile in belligerent atheism, this is not overly surprising.)

Gladwell identified the following three rules of epidemics (in terms of the spread of ideas):

  1. The Law of the Few: There are a small group of people who make a big difference to the proliferation of an idea. The mavens accumulate knowledge and know a lot about the area. The connectors are the gregarious and sociable people who know a lot of other people and, in Gladwell’s words, “have a gift for bringing the word together”. The final type of people are salespeople or (more palatably) persuaders, the people who convince us that something is a good idea. Gladwell’s thesis is that it is not just about the message, but that the messenger matters.
  2. The Stickiness Factor: Ideas have to be memorable in order to spread effectively so there is something about the specific content of the message that will determine its impact. Content matters.
  3. The Power of Context: We are all heavily influenced by and sensitive to our environment. Context matters.

Dawkins’ meme is a very sticky idea and, while there’s a lot of discussion about the Selfish Gene, we now have the field of memetics and the fact that the word ‘meme’ is used (almost correctly) thousands, if not millions, of times a day. Every time that you’ve seen a prawn running on a treadmill while Yakity Sax plays, you can think of Richard Dawkins and thank him for giving you a word to describe this.

My early impressions of some of the problem with the representation of earlier ideas in CS Ed, as if they are new, makes me wonder if there is a fundamental problem with the stickiness of some of these ideas. I would argue that the most successful educational researchers, and I’ve had the privilege to see some of them, are in fact strong combinations of Gladwell’s few. Academics must be, by definition, mavens, information specialists in our domains. We must be able to reach out to our communities and spread our knowledge – is this enough for us to be called connectors? We have to survive peer review, formal discussions and criticism and we have to be able to argue our ideas, on the reasonable understanding that it is our ideas and not ourselves that is potentially at fault. Does this also make us persuaders? If we can find all of these “few” in our community, and we already a community of the few, where does it leave us in terms of explaining why we, in at least some areas, keep rehashing the same old ideas. Do we fail to appreciate the context of those colleagues we seek to reach or are our ideas just not sticky enough? (Context is crucial here, in my opinion, because it is very easy to to explain a new idea in a way that effectively says “You’ve been doing it wrong all these years. Now fix it or you’re a bad person.” This is going to create a hostile environment. Once again, context matters but this time it is in terms of establishing context.)

I wonder if this is compounded in Computer Science by the ability to separate theory from practice, and to draw in new practice from both an educational research focus and an industrial focus? To explain why teamwork actually works, we move into social constructivism and to Vygotsky, via Ben-Ari in many cases, Bandura, cognitive apprenticeship – that’s an educational research focus. To say that teamwork works, because we’ve got some good results from industry and we’re supported by figures such as Brooks, Boehm and Humphrey and their case studies in large-scale development – that’s an industrial focus. The practice of teamwork is sticky, that ship has sailed in software development, but does the stickiness of the practice transfer to the stickiness of the underlying why? The answer, I believe, is ‘no’ and I’m beginning to wonder if a very sticky “what” is actually acting against the stickiness of the “why”. Why ask “why?” when you know that it works? This seems to be a running together of the importance of stickiness and the environment of the CS Ed researcher as a theoretical educationalist, working in a field that has a strong industrial focus, with practitioner feedback and accreditation demands pushing a large stream of “what do to”.

It has been a thoughtful week and, once again, I admit my novice status here. Is this the real problem? If so, how can we fix it?

 

Making Time For Students

Posted: October 20, 2012 | Author: | Filed under: Education | Tags: , , , , , , , , , | 1 Comment

I was reminded of my slightly overloaded calendar today as students came and went throughout the day, I raced in and out of project meetings and RV and I worked on some papers that we’re trying to get together for an upcoming submission date in the next few months. I wish I could talk about the research but, given that it will all have to go into peer review and some of the people reading this may end up being on those panels, it will all have to wait until we get accepted or it comes back on fire with a note written in blood saying “Don’t call us…”

For those following the Australian Research scene, you might know that the Australian Federal Government had put a hold on releasing information on key research funding schemes and that this has led to uncertainty for those people whose salaries are paid by research grants. Why is this important in a learning and teaching blog? Because the majority of Higher Education academics are involved in research, teaching and administration but it’s not too much of a generalisation to say that those who are the most successful have substantial help on the research front from well-established groups and staff who are paid to do research full-time.

Right now, as I write this, our postdoc (RV) is reviewing the terminology of certain aspects of the discipline to allow us to continue our research. RV is running citation analyses, digging through papers, peering at my scrawl on the whiteboard and providing a vital aspect to the project: uninterrupted dedication to the research question. I’m seeing students, holding meetings, dealing with technical problems, worrying about my own grants, preparing for a new course roll-out on Monday… and writing this. RV’s role is rapidly becoming critical to my ability to work.

There are thousands of dedicated researchers like RV across Australia and it is easy to quantify their contribution to research, but easy to overlook their implicit benefit in terms of learning and teaching. Every senior academic who is involved in research and teaching will most likely only still be teaching because they someone to carry on the research and maintain the focus and continuity that only comes from having one major area to work on.

I think of it in terms of gearing. When I’m talking to other researchers, I use one set of mental gears. Inside my own group, I use another because we are all much more closely aligned. I use a completely different set when I talk to students and this set varies by year level, course and student! Making time for students is not just a case of having an hour in my calendar. Making time for students is a matter of making the mental space for a discussion that will be at the appropriate level. It’s having enough time to have a chat rather than a rapid-fire exchange. I don’t always succeed at this because far too many of my students apologise to me for taking up my time. Argh! My time is student time! It’s what I get a good 40% of my salary for! (Not that we’re counting. Like most academics, when asked what percentage of my time I spent on the three areas of research, teaching and admin, I say 50,50,40. 🙂 )

Now I am not, by any means, a senior academic and I am very early on in this process, so you can imagine how important those research staff are going to be in keeping projects going for senior staff who are having to make those gear changes at a very rapid speed across much larger domains. Knowledge workers need the time and headspace to think and switching context takes up valuable time, as well as tiring you out if you do it often enough.

On that basis, the recent news that the Government is unfreezing the medical research schemes and at least some of the major awards for everyone else is good news. My own grant in this area is highly unlikely to get up – my relief is not actually for myself, here – but we are already worried about an increased rate of departure for those researchers who are concerned about having a job next year and are, because of their skills and experience, highly mobile. The impact of these people leaving will not just be felt in terms of research output, which has a multi-year lag, but will be felt immediately wherever learning and teaching depended upon someone having the time and mental space to do it, because they had a member of the research staff supporting their other work. Universities are a complex ecosystem and there are very important connections between staff in different areas and areas of focus that are not immediately apparent when you make the simplistic distinction of staff (professional and academic) and, for academics, research/teaching/admin, research/admin, teaching/admin, pure research and pure teaching. The number of courses that I have to teach depends upon the number of staff available to teach, as well as the number of courses and students, and the number of staff (or their available hours) is directly affected by the number of people who help them.

It’s good news that the research funds are starting to unfreeze because it will say to the people who are depending upon grant money that an answer is coming soon. It’s also saying to the rest of us that we can start to think about planning and allocation for 2013 with more certainty, because the monies will be coming at some point.

This, in turn, stops me having to worry about things like contingency plans, who is going to be working with me, and how I will fund research assistants into 2014 because now I have a possibility of a grant, rather than a placeholder in a frozen scheme. This reduces my  current overheads (for a while) and frees up some headspace. With any luck, the next student who walks into my office will not realise exactly how busy I am – and that’s the way that I like it.

Authenticity and Challenge: Software Engineering Projects Where Failure is an Option

Posted: October 17, 2012 | Author: | Filed under: Education | Tags: , , , , , , , , , , , , , , , , , , , , , , | 2 Comments

It’s nearly the end of semester and that means that a lot of projects are coming to fruition – or, in a few cases, are still on fire as people run around desperately trying to put them out. I wrote a while about seeing Fred Brooks at a conference (SIGCSE) and his keynote on building student projects that work. The first four of his eleven basic guidelines were:

  1. Have real projects for real clients.
  2. Groups of 3-5.
  3. Have lots of project choices
  4. Groups must be allowed to fail.

We’ve done this for some time in our fourth year Software Engineering option but, as part of a “Dammit, we’re Computer Science, people should be coming to ask about getting CS projects done” initiative, we’ve now changed our third year SE Group Project offering from a parallel version of an existing project to real projects for real clients, although I must confess that I have acted as a proxy in some of them. However, the client need is real, the brief is real, there are a lot of projects on the go and the projects are so large and complex that:

  1. Failure is an option.
  2. Groups have to work out which part they will be able to achieve in the 12 weeks that they have.

For the most part, this approach has been a resounding success. The groups have developed their team maturity faster, they have delivered useful and evolving prototypes, they have started to develop entire tool suites and solve quite complex side problems because they’ve run across areas that no-one else is working in and, most of all, the pride that they are taking in their work is evident. We have lit the blue touch paper and some of these students are skyrocketing upwards. However, let me not lose sight of one our biggest objectives, that we be confident that these students will be able to work with clients. In the vast majority of cases, I am very happy to say that I am confident that these students can make a useful, practical and informed contribution to a software engineering project – and they still have another year of projects and development to go.

The freedom that comes with being open with a client about the possibility of failure cannot be overvalued. This gives both you and the client a clear understanding of what is involved- we do not need to shield the students, nor does the client have to worry about how their satisfaction with software will influence things. We scaffold carefully but we have to allow for the full range of outcomes. We, of course, expect the vast majority of projects to succeed but this experience will not be authentic unless we start to pull away the scaffolding over time and see how the students stand by themselves. We are not, by any stretch, leaving these students in the wilderness. I’m fulfilling several roles here: proxying for some clients, sharing systems knowledge, giving advice, mentoring and, every so often, giving a well-needed hairy eyeball to a bad idea or practice. There is also the main project manager and supervisor who is working a very busy week to keep track of all of these groups and provide all of what I am and much, much more. But, despite this, sometimes we just have to leave the students to themselves and it will, almost always, dawn on them that problem solving requires them to solve the problem.

I’m really pleased to see this actually working because it started as a brainstorm of my “Why aren’t we being asked to get involved in more local software projects” question and bouncing it off the main project supervisor, who was desperate for more authentic and diverse software projects. Here is a distillation of our experience so far:

  1. The students are taking more ownership of the projects.
  2. The students are producing a lot of high quality work, using aggressive prototyping and regular consultation, staged across the whole development time.
  3. The students are responsive and open to criticism.
  4. The students have a better understanding of Software Engineering as a discipline and a practice.
  5. The students are proud of what they have achieved.

None of this should come as much of a surprise but, in a 25,000+ person University, there are a lot of little software projects on the 3-person team 12 month scale, which are perfect for two half-year project slots because students have to design for the whole and then decide which parts to implement. We hope to give these projects back to them (or similar groups) for further development in the future because that is the way of many, many software engineers: the completion, extension and refactoring of other people’s codebases. (Something most students don’t realise is that it only takes a very short time for a codebase you knew like the back of your hand to resemble the product of alien invaders.)

I am quietly confident, and hopeful, that this bodes well for our Software Engineers and that we still start to seem them all closely bunched towards the high achieving side of the spectrum in terms of their ability to practice. We’re planning to keep running this in the future because the early results have been so promising. I suppose the only problem now is that I have to go and find a huge number of new projects for people to start on for 2013.

As problems go, I can certainly live with that one!

Industry Speaks! (May The Better Idea Win)

Posted: October 16, 2012 | Author: | Filed under: Education | Tags: , , , , , , , , , , , , , , , , , | Leave a comment

Alan Noble, Director of Engineering for Google Australia and an Adjunct Professor with my Uni, generously gave up a day today to give a two hour lecture of distributed systems and scale to our third-year Distributed Systems course, and another two-hour lecture on entrepreneurship to my Grand Challenge students. Industry contact is crucial for my students because the world inside the Uni and the world outside the Uni can be very, very different. While we try to keep industry contact high in later years, and we’re very keen on authentic assignments that tackle real-world problems, we really need the people who are working for the bigger companies to come in and tell our students what life would be like working for Google, Microsoft, Saab, IBM…

My GC students have had a weird mix of lectures that have been designed to advance their maturity in the community and as scientists, rather than their programming skills (although that’s an indirect requirement), but I’ve been talking from a position of social benefit and community-focused ethics. It is essential that they be exposed to companies, commercialisation and entrepreneurship as it is not my job to tell them who to be. I can give them skills and knowledge but the places that they take those are part of an intensely personal journey and so it’s great to have an opportunity for Alan, a man with well-established industry and research credentials, to talk to them about how to make things happen in business terms.

The students I spoke to afterwards were very excited and definitely saw the value of it. (Alan, if they all leave at the end of this year and go to Google, you’re off the Christmas Card list.) Alan focused on three things: problems, users and people.

Problems: Most great companies find a problem and solve it but, first, you have to recognise that there is a problem. This sometimes just requires putting the right people in front of something to find out what these new users see as a problem. You have to be attentive to the world around you but being inventive can be just as important. Something Alan said really resonated with me in that people in the engineering (and CS) world tend to solve the problems that they encounter (do it once manually and then set things up so it’s automatic thereafter) and don’t necessarily think “Oh, I could solve this for everyone”. There are problems everywhere but, unless we’re looking for them, we may just adapt and move on, instead of fixing the problem.

Users: Users don’t always know what they want yet (the classic Steve Jobs approach), they may not ask for it or, if they do ask for something, what they want may not yet be available for them. We talked here about a lot of current solutions to problems but there are so many problems to fix that would help users. Simultaneous translation, for example, over telephone. 100% accurate OCR (while we’re at it). The risk is always that when you offer the users the idea of a car, all they ask for is a faster horse (after Henry Ford). The best thing for you is a happy user because they’re the best form of marketing – but they’re also fickle. So it’s a balancing act between genuine user focus and telling them what they need.

People: Surround yourself with people who are equally passionate! Strive of a culture of innovation and getting things done. Treasure your agility as a company and foster it if you get too big. Keep your units of work (teams) smaller if you can and match work to the team size. Use structures that encourage a short distance from top to bottom of the hierarchy, which allows for ideas to move up, down and sideways. Be meritocratic and encourage people to contest ideas, using facts and articulating their ideas well. May the Better Idea Win! Motivating people is easier when you’re open and transparent about what they’re doing and what you want.

Alan then went on to speak a lot about execution, the crucial step in taking an idea and having a successful outcome. Alan had two key tips.

Experiment: Experiment, experiment, experiment. Measure, measure, measure. Analyse. Take it into account. Change what you’re doing if you need to. It’s ok to fail but it’s better to fail earlier. Learn to recognise when your experiment is failing – and don’t guess, experiment! Here’s a quote that I really liked:

When you fail a little every day, it’s not failing, it’s learning.

Risk goes hand-in-hand with failure and success. Entrepreneurs have to learn when to call an experiment and change direction (pivot). Pivot too soon, you might miss out on something good. Pivot too late, you’re in trouble. Learning how to be agile is crucial.

Data: Collect and scrutinise all of the data that you get – your data will keep you honest if you measure the right things. Be smart about your data and never copy it when you can analyse it in situ.

(Alan said a lot more than this over 2 hours but I’m trying to give you the core.)

Alan finished by summarising all of this as his Three As of Entrepreneurship, then why we seem to be hitting an entrepreneurship growth spurt in Australia at the moment. The Three As are:

  • Audit your data
  • Having Audited, Admit when things aren’t working
  • Once admitted, you can Adapt (or pivot)

As to why we’re seeing a growth of entrepreneurship, Australia has a population who are some of the highest early adopters on the planet. We have a high technical penetration, over 20,000,000 potential users, a high GDP and we love tech. 52% of Australians have smart phones and we had so many mobile phones, pre-smart, that it was just plain crazy. Get the tech right and we will buy it. Good tech, however, is hardware+software+user requirement+getting it all right.

It’s always a pleasure to host Alan because he communicates his passion for the area well but he also puts a passionate and committed face onto industry, which is what my students need to see in order to understand where they could sit in their soon-to-be professional community.

Dealing with Plagiarism: Punishment or Remediation?

Posted: October 15, 2012 | Author: | Filed under: Education | Tags: , , , , , , , , , , , , , , , , , , , , , , | 6 Comments

I have written previously about classifying plagiarists into three groups (accidental, panicked and systematic), trying to get the student to focus on the journey rather than the objective, and how overwork can produce situations in which human beings do very strange things. Recently, I was asked to sit in on another plagiarism hearing and, because I’ve been away from the role of Assessment Coordinator for a while, I was able to look at the process with an outsider’s eye, a slightly more critical view, to see how it measures up.

Our policy is now called an Academic Honesty Policy and is designed to support one of our graduate attributes: “An awareness of ethical, social and cultural issues within a global context and their importance in the exercise of professional skills and responsibilities”. The principles are pretty straight-forward for the policy:

  • Assessment is an aid to learning and involves obligations on the part of students to make it effective.
  • Academic honesty is an essential component of teaching, learning and research and is fundamental to the very nature of universities.
  • Academic writing is evidence-based, and the ideas and work of others must be acknowledged and not claimed or presented as one’s own, either deliberately or unintentionally.

The policy goes on to describe what student responsibilities are, why they should do the right thing for maximum effect of the assessment and provides some handy links to our Writing Centre and applying for modified arrangements. There’s also a clear statement of what not to do, followed by lists of clarifications of various terms.

Sitting in on a hearing, looking at the process unfolding, I can review the overall thrust of this policy and be aware that it has been clearly identified to students that they must do their own work but, reading through the policy and its implementation guide, I don’t really see what it provides to sufficiently scaffold the process of retraining or re-educating students if they are detected doing the wrong thing.

There are many possible outcomes from the application of this policy, starting with “Oh, we detected something but we turned out to be wrong”, going through “Well, you apparently didn’t realise so we’ll record your name for next time, now submit something new ” (misunderstanding), “You knew what you were doing so we’re going to give you zero for the assignment and (will/won’t) let you resubmit it (with a possible mark cap)” (first offence), “You appear to make a habit of this so we’re giving you zero for the course” (second offence) and “It’s time to go.” (much later on in the process after several confirmed breaches).

Let me return to my discussions on load and the impact on people from those earlier posts. If you accept my contention that the majority of plagiarism cheating is minor omission or last minute ‘helmet fire’ thinking under pressure, then we have to look at what requiring students to resubmit will do. In the case of the ‘misunderstanding’, students may also be referred to relevant workshops or resources to attend in order to improve their practices. However, considering that this may have occurred because the student was under time pressure, we have just added more work and a possible requirement to go and attend extra training. There’s an old saying from Software Development called Brook’s Law:

“…adding manpower to a late software project makes it later.” (Brooks, Mythical Man Month, 1975)

In software it’s generally because there is ramp up time (the time required for people to become productive) and communication overheads (which increases with the square of the number of people again). There is time required for every assignment that we set which effectively stands in for the ramp-up and, as plagiarising/cheating students have probably not done the requisite work before (or could just have completed the assignment), we have just added extra ramp-up into their lives for any re-issued assignments and/or any additional improvement training. We have also greatly increased the communication burden because the communication between lecturers and peers has implicit context based on where we are in the semester. All of the student discussion (on-line or face-to-face) from points A to B will be based around the assignment work in that zone and all lecturing staff will also have that assignment in their heads. An significantly out-of-sequence assignment not only isolates the student from their community, it increases the level of context switching required by the staff, decreasing the amount of effective time that have with the student and increasing the amount of wall-clock time. Once again, we have increased the potential burden on a student that, we suspect, is already acting this way because of over-burdening or poor time management!

Later stages in the policy increase the burden on students by either increasing the requirement to perform at a higher level, due to the reduction of available marks through giving a zero, or by removing an entire course from their progress and, if they wish to complete the degree, requiring them to overload or spend an additional semester (at least) to complete their degree.

My question here is, as always, are any of these outcomes actually going to stop the student from cheating or do they risk increasing the likelihood of either the student cheating or the student dropping out? I complete agree with the principles and focus of our policy, and I also don’t believe that people should get marks for work that they haven’t done, but I don’t see how increasing burden is actually going to lead to the behaviour that we want. (Dan Pink on TED can tell you many interesting things about motivation, extrinsic factors and cognitive tasks, far more effectively than I can.)

This is, to many people, not an issue because this kind of policy is really treated as being punitive rather than remedial. There are some excellent parts in our policy that talk about helping students but, once we get beyond the misunderstanding, this language of support drops away and we head swiftly into the punitive with the possibility of controlled resubmission. The problem, however, is that we have evidence that light punishment is interpreted as a licence to repeat the action, because it doesn’t discourage. This does not surprise me because we have made such a risk/reward strategy framing with our current policy. We have resorted to a punishment modality and, as a result, we have people looking at the punishments to optimise their behaviour rather than changing their behaviour to achieve our actual goals.

This policy is a strange beast as there’s almost no way that I can take an action under the current approach without causing additional work to students at a time when it is their ability to handle pressure that is likely to have led them here. Even if it’s working, and it appears that it does, it does so by enforcing compliance rather than actually leading people to change the way that they think about their work.

My conjecture is that we cannot isolate the problems to just this policy. This spills over into our academic assessment policies, our staff training and our student support, and the key difference between teaching ethics and training students in ethical behaviour. There may not be a solution in this space that meets all of our requirements but if we are going to operate punitively then let us be honest about it and not over-burden the student with remedial work that they may not be supported for. If we are aiming for remediation then let us scaffold it properly. I think that our policy, as it stands, can actually support this but I’m not sure that I’ve seen the broad spread of policy and practice that is required to achieve this desirable, but incredibly challenging, goal of actually changing student behaviour because the students realise that it is detrimental to their learning.

Thoughts on the Fauxpology

Posted: October 14, 2012 | Author: | Filed under: Education | Tags: , , , , , , , , , , , , , , , , , , | Leave a comment

We’ve had some major unpleasantness in the Australian political sphere recently and, while I won’t bore you with the details, a radio announcer has felt it necessary to apologise for a particularly unpleasant comment that he made about the Prime Minster, and the recent death of her father. It was not, I must say, either the most heartfelt or actually apologetic apology that has ever been delivered and the Prime Minster, who quite rightly has better things to do, has chosen not to take this man’s personal phone call for an apology. And, of course, neither should she feel that she has to. Let me state this in plain terms: the offender does not gain the right to demand the way in which an apology is presented, if they wish to proffer an apology. However, let me cut to the chase (for once) and say that an apology without a genuine sense that you have done something wrong, for which an apology is deserved and that will change your behaviour in future, is worthless.

In this case, the broadcaster has previously apologised for remarks, including that the legally elected and sitting Prime Minister of Australia be put in a ‘chaff’ bag and thrown out to sea. However, his apology for the chaff bag comment may have to be scrutinised, in light of what happened at the dinner function at which he made further deliberately offensive and unsubstantiable claims.  At this event he, in between scurrilous remarks, signed a jacket made out of, you guessed it, chaff bags. Therefore, at least in the chaff bag case, it would appear that his previously apology was without conviction and possible not heartfelt: hence, worthless. He did not feel genuine regret or change his behaviour. In fact, if anything, he was now extending his behaviour and disrespect by aligning his signature with a physical representation of his statements.

When public figures mouth the words of regret, yet do not change or feel regret, we are in the territory of what has been neologised as the fauxpology. (Wikipedia refers to this as the Non-apology apology, if it has the form of an apology but does not actually express the expected contrition.) Let me give you some example words (not from said broadcaster I hasten to add):

“My recent comments may have offended some people and, if they did, then I wish to apologise.”

You are not sorry for the action, but you are sorry only because someone has taken offence, or your actions have been uncovered. Ultimately, the idea here is to say ‘sorry’ in such a way that it appears that you have sought, and may be granted, forgiveness without having to actually express responsibility. Of course, if you aren’t responsible for the problem and can move this to being the problem of the people that you’ve offended, then why should you change your behaviour at all? The example above is an “If apology”, where you are only apologising on a conditional basis. Other fine examples include such delightful phrases as “Mistakes were made” because, of course, one is studiously avoiding saying who made the mistakes.

The major problem with the fauxpology is that it is effectively a waste of time. Without a genuine desire to actually avoid the problematic behaviour, the only thing that may change is that the offender is more careful not to get caught. What bothers me from an educational sense is how pervasive these unpleasant non-apologies are.

I have too many students who feel that some sort of fauxpology, where they are sorry that an action has occurred but it is mysteriously not connected to them, is going to make things all better. I’m pretty sure that they haven’t learned it from me because I try to be honest in my apologies and then change things so that it doesn’t happen again. Am I always up to that standard? I’m probably pretty close and I strive to be better at it – but then again, I strive not to be a schmuck and sometimes that doesn’t work either. This separation of responsibility from outcome is a dangerous disconnection. It is most definitely someone’s responsibility if work didn’t get handed in on time and, while there are obvious exceptions and the spirit of charitable interpretation is still alive and well, a genuine recognition of whose responsibility it is leads one towards self-regulation far better than thinking of the work as something that is associated by accidental proximity rather than deliberate production.

I’m lucky in that I rarely expect my students to do anything where they feel they should be contrite (although there are examples, including being rude or disrespectful to their peers, although I wouldn’t push them all the way to guilt on that) but apologising for something as a recognition that whatever it was is both undesirable and now something to be avoided is essential, when you are actually at fault. But it has to be genuine or there is no point. I loathe being lied to so a false apology, especially when immediately backed up by recidivism, is a great disappointment to me.

My students are responsible for their work. I am responsible for their programs, assessment, and ensuring that they can achieve what is required in a fair and equitable environment. If I get it wrong, then I have to admit it and change behaviour. Same for the students. If something has gone wrong, then we need to work out who was responsible because we can then work out who needs to change things so it doesn’t happen again. This isn’t about ascribing punishment or blame, it’s about making things work better. The false apology, like foolish punishment, is easy but useless. As an example. I cannot think of a more useless punishment than writing lines on a blackboard, especially as the simple mechanics of this action lends itself to a deconstruction of the sentence into a form where the meaning is lost by the fifth time you’ve written “I will not challenge the ontological underpinnings of reality” but have really written “I I I I I I …” “will will wll wll wl wl” and getting steadily more squiggly. But this is useless because it is not really tied to the original offence (whatever it happens to be – talking in class, making fart sounds, shuffling the desk) and it has no teaching value at all. This punishment is the equivalent of the fauxpology in many ways: it looks like it’s doing something but not only does it not achieve its aims, it actually works against positive alternatives by providing an easy out.

I’m very disappointed by the public figures who recite these empty phrases, because the community and my students learn their empty words and think “If they can get away with it, so can I” and, ultimately, my students can’t. It’s a waste of their very valuable time and, at some stage, may lead to problems for the vast majority when someone demands more than a fauxpology and there is no real character substance to provide.

Workshop report: ALTC Workshop “Assessing student learning against the Engineering Accreditation Competency Standards: A practical approach. Part 2.

Posted: October 13, 2012 | Author: | Filed under: Education | Tags: , , , , , , , , , , , , , , , , , , , , , | Leave a comment

Continuing on from yesterday’s post, I was discussing the workshop that I went to and what I’d learned from it. I finished on the point that assessment of learning occurs when Lecturers:

  • Use evidence of student learning
  • to make judgements on student achievement
  • against goals and standards

but we have so many other questions to ask at this stage. What were our initial learning objectives? What were we trying to achieve? The learning outcome is effectively a contract between educator and student so we plan to achieve them, but how they fit in the context of our accreditation and overall requirements? One of the things stressed in the workshop was that we need a range of assessment tasks to achieve our objectives:

  • We need a wide variety
  • These should be open-entry where students can begin the tasks from a range of previous learning levels and we cater for different learning preferences and interests
  • They should be open-ended, where we don’t railroad the students towards a looming and monolithic single right answer, and multiple pathways or products are possible
  • We should be building students’ capabilities by building on the standards
  • Finally, we should provide space for student ownership and decision making.

Effectively, we need to be able to get to the solution in a variety of ways. If we straitjacket students into a fixed solution we risk stifling their ability to actually learn and, as I’ve mentioned before, we risk enforcing compliance to a doctrine rather than developing knowledgeable self-regulated learners. If we design these activities properly then we should find the result reduces student complaints about fairness or incorrect assumptions about their preparation. However, these sorts of changes take time and, a point so important that I’ll give it its own line:

You can’t expect to change all of your assessment in one semester!

The advice from Wageeh and Jeff was to focus on an aspect, monitor it, make your change, assess it, reflect and then extend what you’ve learned to other aspects. I like this because, of course, it sounds a lot like a methodical scientific approach to me. Because it is. As to which assessment methods you should choose, the presenters recognised that working out how to make a positive change to your assessment can be hard so they suggested generating a set of alternative approaches and then picking one. They then introduced Prus and Johnson’s 1994 paper “A critical review of Student Assessment Options” which provide twelve different assessment methods and their drawbacks and advantages. One of the best things about this paper is that there is no ‘must’ or ‘right’, there is always ‘plus’ and ‘minus’.

Want to mine archival data to look at student performance? As I’ve discussed before, archival data gives you detailed knowledge but at a time when it’s too late to do anything for that student or a particular cohort in that class. Archival data analysis is, however, a fantastic tool for checking to see if your prerequisites are set correctly. Does their grade in this course correlate with grades in the prereqs? Jeff mentioned a student where the students should have depended upon Physics and Maths but, while their Physics mark correlated with their final Statics mark, Mathematics didn’t. (A study at Baldwin-Wallace presented at SIGCSE 2012 asked the more general question: what are the actual dependencies if we carry out a Bayesian Network Analysis. I’m still meaning to do this for our courses as well.)

Other approaches, such as Surveys, are quick and immediate but are all perceptual. Asking a student how they did on a quiz should never be used as their actual mark! The availability of time will change the methods you choose. If you have a really big group then you can statistically sample to get an indication but this starts to make your design and tolerance for possible error very important.

Jeff stressed that, in all of this assessment, it was essential to never give students an opportunity to gain marks in areas that are not the core focus. (Regular readers know that this is one of my design and operational mantras, as it encourages bad behaviour, by which I mean incorrect optimisation.)

There were so many other things covered in this workshop and, sadly, we only had three hours. I suggested that the next time it was run that they allow more time because I believe I could happily have spent a day going through this. And I would still have had questions.

We discussed the issue of subjectivity and objectivity and the distinction between setting and assessment. Any way that I set a multiple choice quiz is going to be subjective, because I will choose the questions based on my perception of the course and assessment requirements, but it is scored completely objectively.

We also discussed data collection as well because there are so many options here. When will we collect the data? If we collect continuously, can we analyse and react continuously? What changes are we making in response? This is another important point:

If you collect data in order to determine which changes are to be made, tie your changes to your data driven reasons!

There’s little point in saying “We collected all student submission data for three years and then we went to multiple choice questions” unless you can provide a reason from the data, which will both validate your effort in collection and give you a better basis for change. When do I need data to see if someone is clearing the bar? If they’re not, what needs to be fixed? What do I, as a lecturer, need to collect during the process to see what needs to be fixed, rather than the data we collect at the end to determine if they’ve met the bar.

How do I, as a student, determine if I’m making progress along the way? Can I put all of the summative data onto one point? Can I evaluate everything on a two-hour final exam?

WHILE I’m teaching the course, are the students making progress, do they need something else, how do I (and should I) collect data throughout the course. A lot of what we actually collect is driven by the mechanisms that we already have. We need to work out what we actually require and this means that we may need to work beyond the systems that we have.

Again, a very enjoyable workshop! It’s always nice to be able to talk to people and get some really useful suggestions for improvement.

Workshop report: ALTC Workshop “Assessing student learning against the Engineering Accreditation Competency Standards: A practical approach”

Posted: October 12, 2012 | Author: | Filed under: Education | Tags: , , , , , , , , , , , , , , , , , , , | Leave a comment

I was fortunate to be able to attend a 3 hour workshop today presented by Professor Wageeh Boles, Queensland University of Technology, and Professor Jeffrey (Jeff) Froyd, Texas A&M, on how we could assess student learning against the accreditation competency standards in Engineering. I’ve seen Wageeh present before in his capacity as an Australian Learning and Teaching Council ALTC National Teaching Fellowship and greatly enjoyed it, so I was looking forward to today. (Note: the ALTC has been replaced with the Office for Learning and Teaching, OLT, but a number of schemes are still labelled under the old title. Fortunately, I speak acronym.)

Both Wageeh and Jeff spoke at length about why we were undertaking assessment and we started by looking at the big picture: University graduate capabilities and the Engineers Australia accreditation criteria. Like it or not, we live in a world where people expect our students to be able to achieve well-defined things and be able to demonstrate certain skills. To focus on the course, unit, teaching and learning objectives and assessment alone, without framing this in the national and University expectations is to risk not producing the students that are expected or desired. Ultimately if the high level and local requirements aren’t linked then they should be because otherwise we’re probably not pursuing the right objectives. (Is it too soon to mention pedagogical luck again?)

We then discussed three types of assessment:

  • Assessment FOR Learning: Which is for teachers and allows them to determine the next steps in advancing learning.
  • Assessment AS Learning: Which is for students and allows them to monitor and reflect upon their own progress (effectively formative).
  • Assessment OF Learning: Which is used to assess what the students have learned and is most often characterised as summative learning.

But, after being asked about the formative/summative approach, this was recast into a decision making framework. We carry out assessment of all kinds to allow people to make better decisions and the people, in this situation, are Educators and Students. When we see the results of the summative assessment we, as teachers, can then ask “What decisions do we need to make for this class?” to improve the levels of knowledge demonstrated in the summative. When the students see the result of formative assessment, we then have the question “What decisions do students need to make” to improve their own understanding. The final aspect, Assessment FOR Learning, is going to cover those areas of assessment that help both educators and students to make better decisions by making changes to the overall course in response to what we’re seeing.

This is a powerful concept as it identifies assessment in terms of responsible groups: this assessment involves one group, the other or both and this is why you need to think about the results. (As an aside, this is why I strongly subscribe to the idea that formative assessment should never have an extrinsic motivating aspect, like empty or easy submission marks, because it stops the student focussing on the feedback, which will help their decisions, and makes it look summative, which suddenly starts to look like the educator’s problem.)

One point that came out repeatedly was that our assessment methods should be varied. If your entire assessment is based on a single exam, of one type of question, at the end of the semester then you really only have a single point of data. Anyone who has ever drawn a line on a graph knows that a single point tells you nothing about the shape of the line and, ultimately, the more points that yo can plot accurately, the more you can work out what is actually happening. However, varying assessment methods doesn’t mean replicating or proxying the exam, it means providing different assessment types, varying questions, changing assessment over time. (Yes, this was stressed: changing assessment from offering to offering is important and is much a part of varying assessment as any other component.)

All delightful music to my ears, which was just was well as we all worked very hard, talking, discussing and sharing ideas throughout the groups. We had a range of people who were mostly from within the Faculty and, while it was a small group and full of the usual faces, we all worked well, had an open discussion and there were some first-timers who obviously learned a lot.

What I found great about this was that it was very strongly practical. We worked on our own courses, looked for points for improvement and I took away four points of improvement that I’m currently working on: a fantastic result for a three-hour investment. Our students don’t need to just have done assessment that makes it look like they know their stuff, they have to actually know their stuff and be confident with it. Job ready. Able to stand up and demonstrate their skills. Ready for reality.

As was discussed in the workshop, assessment of learning occurs when Lecturers:

  • Use evidence of student learning
  • to make judgements on student achievement
  • against goals and standards

And this identifies some of our key problems. We often gather all of the evidence, whether it’s final grades or Student Evaluations, at a point when the students have left, or are just about to leave, the course. How can we change this course for that student? We are always working one step in the past. Even if we do have the data, do we have the time and the knowledge to make the right judgement? If so, is it defensible, fair and meeting the standards that we should be meeting? We can’t apply standards from 20 years ago because that’s what we’re used to. The future, in Australia, is death by educational acronyms (AQF, TEQSA, EA, ACS, OLT…) but these are the standards by which we are accredited and these are the yardsticks by which our students will be judged. If we want to change those then, sure, we can argue this at the Government level but until then, these have to be taken into account, along with all of our discipline, faculty and University requirements.

I think that this will probably spill over in a second post but, in short, if you get a chance to see Wageeh and Jeff on the road with this workshop then, please, set aside the time to go and leave time for a chat afterwards. This is one of the most rewarding and useful activities that I’ve done this year – and I’ve had a very good year for thinking about CS Education.

Offering a Way Forward: Saying “No” Constructively

Posted: October 11, 2012 | Author: | Filed under: Education | Tags: , , , , , , , , , , , , , , | Leave a comment

I was a shockingly bad undergraduate student. I am rather ashamed of this, in hindsight, because it could have been the most dreadful squandering of the opportunities that had been made available to me. Fortunately, intervention at a later time allowed me to recover from this mis-step and I arrived at the lofty heights that you know now: a semi-anonymous blogger with logorrhoea, employed as a low level academic at a good University. 🙂

One of the experiences that sticks in my mind, and I believe I’ve mentioned it before, was a point when I was refused supplementary examination for a course. It was, to be honest, completely reasonable that my request was refused as I had achieved the lofty score of zero for that examination and, although I was sick on the day, I had done no work at all during that term. There is no argument that my request was anything other than a small part in my continuing quest to scrape by and do as little as possible. However, what I will address is what happened next.

Nothing.

I was in an academic’s office, I had scored zero on the exam and I had not done any work. If someone showed up in my office under those conditions, I’d at least ask “Why?” Ok, sure, you’re not getting a supp from me, because I’m not a soft touch by any stretch of the imagination, but I can’t help feeling that I’m just not doing my job unless I work out what has gone on and whether I need to step in or refer you to someone else. This probably explains why I’m so prone to jumping up and down when people talk about bad students, as if this is a permanent and immutable condition, because while I completely accept that some students (a small group) will resist all efforts to bring them into a culture, I strongly believe that it is far more likely that little effort has been spent to try and recover these students in the first place.

Was the academic right to refuse my application? Yes, no argument, but the methodology sucked. No follow-up. No identification of problems. Did I learn from this? No, I didn’t. I distanced myself from the mark because I had expended no effort – this may sound familiar from my discussion of pedagogical luck from yesterday. The same self-destructive patterns played out until someone (a big group of someones actually) took enough interest in me to make me explain what I was doing, make me realise what I was doing and show me the way forward. (And, for the record, thank you to all my friends and my wife, Katrina, but I’d like to specifically note Cathy and Andrew, where poor Cathy had to basically help me focus while I tried to learn what being a real student was about. And thanks to KJM for suggesting that I start doing the new degree in the first place.)

The reason that this is back in my mind is that our Summer Research Scholarships are being advertised at the moment, where students come and work on a research project for 6 weeks over Summer with a small stipend. Competition for these scholarships is surprisingly fierce and I am lucky enough to be offering some interesting projects, both in networking and education research, that people find attractive. Most of the students who have applied for my support have had a level of interest and academic achievement that I can easily support them in what is a highly competitive environment (University ranking systems, not mine, I hasten to add). However, one of the applicants was not all that competitive, so I had to say that I really couldn’t support them in their application. Yet. But here are some ways to be ready for this next year. What was the problem for this year? How are you going at the moment? Ok, keep working on this and I look forward to seeing you come back to me next year, but here’s another suggestion.

The way of saying “No” is important, and I believe that this student can reach the required level, they’re just not there yet. So we discussed how they could improve their overall eligibility, including coming up with their own ideas to fit in with their thoughts on research, so that when they come back next year, they come back not only more eligible but with a strong idea of what they want to do with us. I expect to see this student back next year because I’ve done similar things before and, more often than not, the student comes back: purposeful, more focused and much, much more ready to undertake the kind of projects involved. I have already loosely allocated some resources in case the student does what I’ve suggested for improvement in the shorter term. Next time, I hope that I’m in a position where I must support this student’s application but I can only claim to have helped here if I’ve taken the time to think about what is going on and how I might be able to help.

In the spirit of charity, I must confess that I have no idea if the lecturer, who didn’t ask me about what I was up to, was just having a bad day or if my performance was so poor that he just decided that I was one of the small number who wasn’t ever going to come good. You can’t judge someone on a single event or interaction and it would be ridiculous to not admit to the human failing of the ‘bad day’. However, my overall experience, as a bad student, was that I was not worth wasting time on. I was not worth the risk. And, yet, no-one ever asked me if there was a reason for it or why I was doing it, or if there was something that I needed help with.

I’m happy to say that I believe that a culture like that is now, rightly, judged and found wanting. That people are far more likely to ask why and suggest how or who to see next. But it’s not universal. We may not always achieve universality, people are people and bad days are bad days, but the first step, for me, is looking at how we say “No.”

 

Moral Luck and Voluntary Action: Is There a Corresponding Pedagogical Luck?

Posted: October 10, 2012 | Author: | Filed under: Education | Tags: , , , , , , , , , , , , , , , | Leave a comment

Moral luck (sometimes described as moral accident) describes a situation where someone is assigned moral blame or praise for something happening, even though the person was either not in control of what was happening or could not affect its consequences. There are many examples, including the traffic accident scenario described in the Moral Luck link, and there are several different classifications of moral luck but let me focus on one: the situation where you either take no positive steps to address a situation, or actively take negative steps, yet the outcome is still positive. To a consequentialist, this is a beneficial outcome and constitutes an example of Resultant Moral Luck. One of the most extreme examples is that you randomly stick your foot out, hoping to trip someone in the street, and accidentally bring down a criminal being pursued by the police. The outcome is good, you are possibly a hero, but any assignation of a moral intention to your actions is deeply flawed: you weren’t in control of the situation, you did not intend the outcome and, in fact, you had hoped to cause harm. The voluntary action that you took was in no way intended to cause this outcome. Yet, you are a hero.

When we look at methods and practices of teaching, it quickly becomes apparent that there are many different approaches and, upon doing some reading, that these have different utilities and efficacies. Your choice of pedagogy is rarely a one-size-fits-all approach and, especially if your institution takes a relatively traditional approach, we have to start to wonder exactly which part of the evolutionary selection stage we are in. Have we, by chance and/or design, arrived at an elegant and efficient design years ago that cannot be improved upon by recent findings or are we ripe for new development, new directions and entirely different ways of teaching?

I would argue that, if we are not taking steps to confirm where we are in the developmental timeline or we are not taking steps to examine what we do with the intention of improving,  then we are wandering in an area that we could call pedagogical luck, where any positive teaching outcomes that may arise cannot be attributed to our voluntary actions and intentions. Are we in the territory that Feynman was referring to when he quoted Gibbon:

“The power of instruction is seldom of much efficacy except in those happy dispositions where it is almost superfluous”?

The notion of pedagogical luck, much like moral luck, raises questions of responsibility and accountability. It also explains how we can misattribute blame, because we risk not having a clear ethical framework that can ascribe intention, action and outcome in a meaningful way. In moral luck two people can speed through the same red light, yet only one causes an accident because a child runs into an intersection and the other one may receive a fine for running the light. The outcomes could almost not be any more starkly different: in one a human is injured or killed, the other is a purely administrative outcome. We certainly will attribute more blame to the first driver than the second, despite the fact that both had no desire to kill, nor did they act any differently – the reason that this is resultant is because this is just the way things turned out.

If an academic works with a class and, as it happens, everyone passes, then we would usually assume some intention and voluntary action was involved on the part of that academic. The outcome, for the students and the academic, are both beneficial. It is… unlikely… that said academic would then walk around stating “I’m amazed that they all passed – I barely even showed up to class and I didn’t revise the notes.” However, where someone has done nothing (or has not taken a voluntary action to cause change) and all of the students fail, we can expect (with a reasonable certainty) that the usual statements of blame shifting may start to occur: the students were stupid, lazy, unprepared, insufficiently attentive, the material was pitched at the right level but the students didn’t work hard enough, et cetera. “I have kept the course the same, it is obviously the students who are at fault.” Of course, it’s easy to see why if we have not taken any active steps to change anything – why should we be held responsible for an action that has had either neutral or positive outcomes in the past? Why should we judge the killer-driver any more harshly than the red-light-runner? The outcome is a matter of luck.

This is highly undesirable behaviour so how can we avoid the issues involved in depending upon pedagogical luck? I’m tempted to delve into virtue ethics here and argue that, of all places, that if you can’t find a virtuous seeker of knowledge in a University then perhaps we should all go back to a simple agrarian existence and wait to die of some horrifically mutated bovine disease that we no longer have the wit or wisdom to cure. However, I suspect that we don’t need to all be virtuous, all the time, to adopt a simple maxim that commits us to seeking improvement in our learning and teaching, or to confirming that our approaches are still valid. Where possible, such endeavours should be public and shared widely, so that our lessons can be learned elsewhere. Yes, we’ve wandered fairly heavily into Kant because I’m effectively arguing good will as a stand-alone virtue, regardless of what is achieved. In the absence of a guarantee of virtuous people, and we all have bad days, then perhaps it is a commitment to scholarship, review and reflection that can allow us to take that fresh approach to pedagogical development and implementation that will cause us to be less susceptible to blame shifting where it is inappropriate and less likely to form cargo-cultish ideas as to why certain courses are succeeding or failing.

It is a simple idea: claiming beneficial outcomes as caused by us when we have done nothing is questionable, ethically, especially when we refuse to accept negative outcomes under the same scenario. By identifying that pedagogical luck is possible and readily identifiable in certain practices around the world, we clearly identify the need to avoid the situations where it can dominate.

← Older Entries
Newer Entries →