MIKE: Measurement Is the Key to Everything

One of the problems in convincing other people to try alternative learning and teaching approaches is that, basically, everyone is as busy as you are. While you might not accept that, you might be the busiest person in the Universe, then perhaps you can accept that everyone thinks that they are as busy as you are. In a world where academics struggle to fit in research, administration, teaching, marking, personal development, grant applications – oh, and their real lives – it’s not surprising that a lot of first reactions to ‘have you tried something new’ is ‘do you know how busy I am?’

We have a big advantage in ICT in that most people are very open to the scientific method of measurement, analysis and evaluation (potted version). So why is it that when someone says “Have you tried this” and you ask something in return like “Well, no, but how long will it take and what will be the benefits?” you’ll be lucky to get an answer to the first half of that compound statement, let alone the benefits. “Your students will be happier” is very hard to quantify – but “we reduced our drop-out rate by 30%” is a cold hard fact. (Well, it purports to be. If someone has done their due diligence, it’s a fact.)

There is, of course, a problem. In order to be able to assess the impact of what we’re doing, we have to establish the baseline (how things were before we started), apply our changes, measure the outcomes and then try and determine if what we did had anything to do with the perceived change or whether it was all random noise. This is not helped by the fact that a lot of classes are small, smaller than we need for statistical validity, or that we can’t easily establish cohorts of the right size or consistency. But, being honest, the first problem for many people is that they do not even think of measuring the impact of what they’ve done until after they’ve done it.

So here’s the first of my three slogans.

MIKE: Measurement Is the Key to Everything.

Looking at that model that I keep discussing, I have three separate places to “lose” knowledge in its flow to my students (to reduce the efficacy of flow). The first is in the teaching process itself. If I don’t have the knowledge, I can’t pass it on. If I choose not to share the knowledge, I can’t pass it on. Next, the medium of exchange (that disconnected external transfer from teacher to learner) will make a difference. If I write everything I know in a book and give it to my class, tell them that the final exam is in two months and walk off – I’m in a high-loss environment. So the medium can and does make a difference but it can only facilitate knowledge transfer by minimising the loss or maximising availability of learners to knowledge. It can’t add knowledge. Finally, what the learners themselves do will have a big impact on how the knowledge is processed and assimilated. That’s why, even without curve grading, those Bell curves seem to show up so frequently – in a similar teaching environment, with the same lecturer, individual students still have some variation. We can, of course, vary the peak of the curve but we would expect to see some variation in an otherwise identical environment. A lot of this has to do with the environment that students had before they reached us, which is interesting if only for the fact that this medium of knowledge transfer may now appear to have both memory and temporal aspects – perhaps our dealing with this previous environment, or accepting that such differences exist, in the construction of our transfer medium is as important as the knowledge that we bring to the situation.

Now I can quantify the effort that I put in to my teaching activities, if I’m honest with myself and count time spent actively creating new approaches or materials – and discounting those times I spend in the tea room pontificating about things I never apply. (I don’t think that such sessions have no value, but I hesitate to count them in a genuine measurement of producing new teaching materials unless I am actively mentoring or I run off and do something with that. Even then, I discount the time for each coffee I had. 🙂 ) If I have assessed the student quality or class metric that I want to change, and I have established a baseline on the cohort (somehow), I can come up with an measurement of time spent, or difficulty level to surmount, to implement my new approach and I can then present the effort, and the outcome, along with the environment in order to show other people what I did and how they could do the same thing.

I recently made some changes to a new first year course and I was fortunate in that I achieved a much higher pass rate than usual for the effort that I expended, with excellent process awareness of how to correctly design and finish programming projects on time. Hooray, you might think. Aha – I had only 21 students (it was the first offering and the pipeline was barely filled) and these students had, in the main, correctly self-selected as having programming experience before coming to University. Yes, we had a good result, excellent engagement, and high participation and we achieved it with the standard load model for writing a new course but our environment was not the standard one. Next semester, when I have 130 students from across the range of the intake, I will have an environment where, when I measure how many hours I spent on each activity, I will have much more applicable environment to realistic teaching situations in other Australian Universities.

I’ll be able to assess each student’s early indications of prowess, from their marks in other courses, and compare them to what is achieved in this new course. I can then start to make statements indicating what the benefits of the approach are. But, to do that, I have to think measurement from the moment I start working on the course, keep track of my time, note where I make changes, look for which factors are being affected and, finally, be honest if I can see trends but not significance, an indication of a Bayesian model but not a confirmation. I have to think about quantitative and qualitative assessment mechanisms – I may have to get surveys pre-approved or start designing custom assessment forms. I have to think about how I am going to be able to assess the worth of what I’ve done in the ground-up design of this course weeks before Week 1 – not only for my own benefit, but for communication with others and for possible papers or presentations.

Ultimately, I can give you a warm feeling and tell you that ‘students will love this’ or I can show you the well-written, thoughtful and mature advice on process improvement for timely completion of software projects, well proof-read and easy to read, that I received from the vast majority of the students that I had in my course – after they’d been in the system for less than 12 months. And I could tell you how much effort that took, and the caveats of the environment, and then, with all of those caveats, you might think about how you could do a similar thing in order to achieve a similar result. Or to see if I’m barking mad. That is, after all, what we expect our students to do: assemble evidence, weigh and analyse, complete the evaluation and come to a conclusion. Then act.

Measurement: it really Is the Key to Everything.