Who’s better, who’s best?

I’ve mentioned earlier that I assemble my final grades out of a number of assignments, all designed to test different areas, where a range of marks are possible. There are very few 0 or 1 situations in the assignments, with the possible exception of participation marks (and these never make up a big percentage of anything). Now the natural spread of marks across all of these assignments contributes to a slightly noisy final mark. As a general indicator of performance, the band into which your final mark falls is going to be pretty reliable. If you get 50-64 in one of my classes then you know enough to move on from the course. You have enough knowledge. If you get 45-49… well, then we offer you a chance to sit the exam again or, sometimes, do more assignment work, because we’re sitting in a slightly noisy zone and giving you another chance might get you over the edge.

But what is the real difference between a student who gets 50 and one who gets 60? 64 and 65? 98 and 99? Which of these leap out to you and say “Well, obviously, this student is better!” What do we even mean by better  in this case? In my opinion, if a course is designed correctly, and the students are properly prepared, then a pass mark is available for everyone if they do the work and apply themselves. The higher grades are only open to people who move on to synthesis, or can demonstrate deeper understanding, or can communicate their ideas more clearly, or… you get the idea. It’s often a sign of increased effort, to a large extent, than greater ability. (I’m sure we all have very smart students who are mildly bemused as they fail again because they just didn’t hand enough up or actually read a book.) In terms of general bracketing, a student who achieves a credit has probably done more than one who has a pass.

Okay, yes, the borders are tricky but with enough assessment work and opportunity, sometimes people just don’t make it over the line. If we have the ability to fail people, then we can apply some sort of higher level banding to higher level concepts. (There’s lots to say here and I’m trying to keep this short and on point, but I’ll try to return to this later.)

When it comes to numerical marks, though, inside the bands, that’s a bit shakier for me. Yes, there’s a difference between 100 and 85 (do we need a VHD?), but 99 and 98? No, they’re pretty much the same. It’s a shame, therefore, that we often rank students in our heads on these numbers rather than their grades. Or we average the grades and come up with a number that is as accurate and as useful as an average can be. Should we be using the average? Mode? Median? Harmonic mean? What makes sense in the face of what these students have done?

It’s a shame then that values like these, or averages of these, or GPAs, have so much weight in the community. It’s not as if most people dig down, or have the ability to dig down, into the underlying courses to see what these things actually mean. Perhaps we should be awarding marks for mastery of core materials, a separate mark for performance in projects and yet another mark for advanced or extension work, tying it into industry practices and terminology? Then you could look at the numbers, as they are, and having isolated the components that have been jammed together before, get an idea of what that number means.

Imagine that a student has a perfect GPA for a set of courses that, as it turns out, have nothing to do with what your company does – except that the name sounds about right. Another student with a slightly worse GPA has perfect marks for core, including the language and techniques you want, and is fantastic on projects. You may find this out at interview or in the application but, if you only take perfect GPAs and ditch everything else, you’ll never even see number 2.

I’m not saying that our current system is unworkable, because it obviously works, but I have started to wonder what else we could do – in an ideal world. There’s no solution here, this is not amazingly pragmatic thinking, which doesn’t take into account the fact that so many people who take our students want that number. But that doesn’t mean we can’t talk about it and think about it. Maybe there’s something better? The first step is to work out what it is.


Bad Summaries Ruin Good Reports: Generation Why?

A media release came around on Friday from Universities Australia called “Generation WhY? (sic) Students question point of science and maths“. You can read the media release, key findings and the associated report here. The key findings, for students who are both STEM and non-STEM, are published with a series of pull quotes and explanations underneath them. For my own purposes, I’ve removed those because I want you to read the key findings in the raw:

  1. More than 40% of students surveyed did not feel encouraged to do well in maths and science by their teachers at high school
  2. 1 in 3 students were influenced by past teachers in their university choices
  3. 1 in 5 STEM students somewhat or totally engage in the stereotype that science is for nerds.
  4. Students sometimes felt (that) STEM subjects do not align with other interests and abilities.
  5. Some students interviewed saw no positive value from pursuing STEM as a career.
  6. An inability to understand or work with the precise black-and-white nature of science, as opposed to less structured processes, turned some students away.

The report itself is 144 pages long but from page 88 it’s an appendix containing the survey so it’s not too long a read. However, those 6 statements above are, well, in the politest way possible, not very precise. Finding 2., for example, has accompanying text that implies the influence was negative – that 1 in 3 students were discouraged, rather than influenced. Finding 3 is interesting but how many non-STEM students feel the same way? Finding 4 – sometimes? Let’s look at the questions to get an idea of how the survey was framed. The initial questions are all basic demography, then we get to the meat.

Question 19. As a person are you primarily?

  • More socially outgoing and like being the centre of attention
  • More a quiet and private person and like being with your own thoughts
  • Not sure/can’t answer

Urm. I’m socially outgoing but I like time alone with my thoughts. I’m sure of that, however. But this is a quibble. Not many people will have a problem with this. Let’s look at another one.

Question 21. As a person do you primarily?

  • Go with your gut instincts
  • Focus on cold hard facts
  • Not sure/Can’t answer

Urm, again. COLD HARD FACTS. M’lud, I think that we’re leading the witness a tad. How about “Go with your instincts/Focus on the facts”? (Still lots of room for improvement)

You can read the rest of the survey yourself – because I don’t want you all to die of boredom. First thing is that, yes, of course, survey design is hard and I’m sure that a lot of thought went into this survey. However, the press release that came out from this survey makes some claims that, if true, mean that we in the higher edu sector are pretty much stuffed in some ways, because we just won’t get the students here to work with in the first place. Once a student gets into STEM, I can work with them. If, as the survey suggests, I’m losing 33% to teacher discouragement, or 40% to not doing well, or 20% to the nerd factor, I’ve lost a vast number of potential students.

Reading the survey, rather than the keypoints, is far more illuminating. It turns out that teacher influence can be either way, which should have been obvious in the summary. It paints teachers in a much fairer manner. That whole ‘science is for nerds’ is in the middle of a question with lots of opinion options and a 5 point rating scale for agreement. So 20% of STEM students ticked the Totally or Somewhat agree box.

Hang on. That means that 80% of the people in STEM either can’t answer or don’t think it’s for nerds. Page 69 of the report talks on this. I quote: “A higher proportion of STEM respondents somewhat agreed with the statement science is for nerds than did non-STEM respondents.”

They then show the results table. 1364 students in total, 730 non-STEM, 634 STEM. 96 of non-STEM thought it was for nerds, 124 of STEM thought it was for nerds. All other results were disagreers. They’ve already removed the can’t answer people from the survey. That’s 13% of outside STEM people and 19.6% of STEM. Now all of these students are currently enrolled, at University, so the people who are more likely to think science is for nerds are already inside our borders. So, the actual finding is:

“Around 1 in 10 students outside of STEM have a negative image of science as being for nerds, and the number increases slightly to just under 1 in 5 for students inside STEM. Overall, roughly 1 in 6 first-year students surveyed have a perception of science as nerdy.”

That’s surprisingly positive to me. I’d always thought that everyone thought we were enormous dorks. Hooray! Checking the figures, only 5% of STEM students totally agree anyway, compared with 3% of non-STEM, but we have a lot more ‘somewhat agrees’ which really drives the numbers up in STEM.

Here’s the quote that was underneath the 1 in 5 figure in finding 3: “Also if you see scientists on the news like, there’s kind of a stereotype that you will see… Like kind of wearing glasses… They never dress well.” That seems pretty damning. Not only do people think we’re nerds, they took the time to write this down.

But that quote doesn’t come from the survey. That pull quote is not from the same source as the survey data, it’s an anonymous student comment from the Phase 1 pre-survey focus group. In fact, there is no text box associated with that question (Question 80) – Question 81 is a question with a text box, but it’s for comments about the survey itself. Associating that quote with that finding makes a very strong implied linkage that is very. simply. not. there. The initial focus group at University of Sydney was composed of 8 people, a 5/3 male/female split, all first-year, with five B.Sc and three B.A. students. What they admitted that they felt about stereotyping was used to build the survey question at the end. But putting their pre-survey thoughts together with a post-survey result is something that, well, ok, maybe it’s done all the time, but I wouldn’t do it myself.

Those two entities have no linkage – unless it is to say “Hey, the focus group thought everyone would think that science was for nerds but they turned out to be wrong – it’s less than 20% on average and we’re harder on ourselves in STEM, about being cool, than other people think we should be. Woo!” because that recognises the data origin and what the result means. The way that it is presented in the key findings is misleading.

Finding 4 is a curious one (Students sometimes felt (that) STEM subjects do not align with other interests and abilities.) because there is a question, Q50, that asks about why you chose a particular degree. However, the report does not clearly show the detail of the responses and the question just lists ‘Best fit for my interests and abilities’ as one of the options for “What are the reasons for your choice of University degree/course/program”. Searching for the words “interests” or “abilities” in the text brings up some earlier quotes and I must be missing something because I couldn’t find anything to support finding 4, beyond a brief quote from the pre-focus group again. The word ‘align’ doesn’t occur in the report. I’ve read all the questions and can’t see where that finding could be derived. I must be missing something because I can’t find a single solid point in the report, or a summary, that supports this key finding. So, dear reader, if you can find it, please help me out and show me where it is! (I’m a bit tired, so forgive me if I’ve missed the obvious.)

I can’t help but feel that this media release, focusing on negative interpretation and using contextualising quotes that reinforce that interpretation, is doing a disservice to the interesting data contained within the report. Check it out for yourself to see how else things have been reported one way in the actual report and then projected out through the media release. If nothing else, it’s a teaching example in itself of how you can present data accurately but in a way that will very definitely channel someone’s interpretation – especially if they don’t bother to read the original article. If you read the report, you can see that the writers are concerned about the statistical validity because only 12% of their target group responded.

It’s a reminder that all the work you put into your survey design and data analysis process is nothing if that message is lost or adulterated in the search for an easy message. The message matters more than the medium. Once again, the medium is important, but the message is paramount.

Finally, it’s a reminder that we always must read the primary source, to at least calibrate the secondary and tertiary reports.


The Pipeline (again) or “Am I really working on Sunday?”

This post will appear on the 15th of January, 04:00am, Adelaide, South Australia daylight saving timezone. Does this mean that I am up at that hour or working on a Sunday morning? Far from it! I’m writing this post on Friday afternoon, as my day winds down.

After I wrote my post on the pipeline, I realised that my commitment to a daily blogging schedule had created a 15-30 minute hole in each of my days and sometimes I don’t have that much time to spare. We’ve all had the days when, after thinking about it for 6 hours, you finally take a well-earned bathroom break. If those 5 minutes are hard to scavenge, where would I get the headspace to write something amusing and amazing on a day like that?

Re-enter the pipeline! I am now working at least a day in advance. This has two advantages, firstly, that I take the pressure off myself and, secondly, that I can adjust my posting times to maximise the chances of being picked up and read from the Education topics page. I’m not writing this for my own benefit (well, not exclusively) – I want this to be read. By having my posts ready a day in advance I can commit them for publication at a time when more potential readers are awake!

Australia is a beautiful place but it’s a long way from anywhere and a large portion of the English-speaking education community are asleep when I’m awake. If you flick on topics/Education at 9:00am every day in San Francisco – I should be in bed because it’s about 3:30am where I am. So I’m experimenting with a 4am posting time to try and catch a sweet spot for the US being awake and the UK not yet being asleep. I wouldn’t be able to do this with live update, or without a pipeline, unless I was wishing to sometimes skip a day’s post to move it into the next hot zone.

I have jumped around a bit in scheduled publication time. I’m reviewing results this weekend* to see which has been the ‘most successful’ viewing time and I’ll start using that as my default publication time. I don’t care how many people ‘follow’ or ‘like’ my posts – but I do care if nobody reads what I’m writing because that means I’m wasting my time.

You can probably tell that I’m not all that keen on wasting my time. Yes, this requires both forethought and discipline but I am finding it incredibly liberating to know that all I have to do between now and Monday is think of one cool thing and put that in the pipeline too. Hmm. I have 5 minutes left…

(*PSST: Measurement is the Key to Everything)

 


THE (Thinking Higher Education).

Well, if you’ve survived the MIKE post and the SWEDE post, you’ve probably guessed that I like acronyms. However, TMASTB (Too Many Acronyms Spoil The Broth), so I’ll present one more and go back to discussing other issues. However, these three acronyms together will drive a lot of what I talk about and, with any luck, I’ll have strangers around the world muttering “MIKE THE SWEDE” at each other.

I’m not holding my breath.

There are so many different ways to educate, some many different environments, students and teachers, choosing an approach that will work can sometimes appear to be an overwhelmingly difficult problem. How do I cater for everyone? How do I deal with large ranges in ability? How can I be fair to all the groups I deal with and, at the same, fair to myself and my own family, giving myself enough time to work and live.

This is where “THE” comes in. I’m not trying to solve every educational problem, I’m thinking about higher education. The students that I deal with have already shown that they can handle education to some extent, whether traditional classroom-based study, collaborative group learning in more experimental frameworks or home-schooling. They have passed sufficient entry requirements to attend classes at my institution. They can read. They can write. They are fundamentally numerate. They may have pre-requisite subject requirements to meet on top of this. For example, all of my students have at least one course of mathematics to the matriculation level. As an educator, I have so much to be grateful for in that a lot of the hard work has already been done by the educators before me, who excited these students about learning, who gave them the basis upon which they could develop to the point that they made it to me. For the families who supported them, or the groups that supported the students or the families when they families couldn’t do it by themselves.

I looked into publishing a book once, on computerising wineries (previous career), and I attended a really interesting workshop on pitching books to publishers. One of the most important pieces of advice was that no book was ever really suitable for ‘all ages from 8 to 80’, so don’t claim it. Work out who your book is for, write it for that group and then advertise it correctly. Who should my courses be suitable for? What do they come in with? What do I want them to leave with? How do I tell people that they will benefit from this course so come and try it?

Thinking Higher Education means thinking about students who have already demonstrated an ability to work within our systems, who have already stuck at education for 12+ years and who most likely perfectly capable of passing our courses, if we keep them engaged, do our jobs properly and their own lives don’t get in the way. Yes, there will be ranges of ability and dedication, but these tend to be smaller than are seen in the early primary or early secondary years. The kids who caused lots of trouble in class probably aren’t here anymore although, with any luck, they’ll sort themselves out in a while and we’ll welcome them back as mature-aged students with open arms. Yes, you’ll have mature-aged students sitting next to 17 year olds and you’ll need to think about that but if they’re sitting in your course then they need (and sometimes even want) you to teach them. Or to give them the right environment in which they can teach themselves. But they all need the same thing and, theoretically, they are on a much tighter track in their quest for degree completion than trying to match the diverse requirements of a group of 15 year olds sitting in an English class.

It’s not impossible. It’s certainly not easy and it doesn’t downgrade the role of a University educator but, despite having students from all over the world, the country, the age groups, the demographic spectrum – we can manage this problem and share our knowledge.

So THE is a positive thing, a reminder that our job is manageable, an appreciation of the work that has been carried out before by our colleagues in schools, a mark of respect to their successes and an awareness that students come to us with a great deal of potential and previous experience, both of which will shape their future.

Putting it together – Measurement, Environmental Awareness and Managing Scale, you get MIKE THE SWEDE.

Happy muttering!

(No more new acronyms for at least a few days, I promise!)

 

 


MIKE: Measurement Is the Key to Everything

One of the problems in convincing other people to try alternative learning and teaching approaches is that, basically, everyone is as busy as you are. While you might not accept that, you might be the busiest person in the Universe, then perhaps you can accept that everyone thinks that they are as busy as you are. In a world where academics struggle to fit in research, administration, teaching, marking, personal development, grant applications – oh, and their real lives – it’s not surprising that a lot of first reactions to ‘have you tried something new’ is ‘do you know how busy I am?’

We have a big advantage in ICT in that most people are very open to the scientific method of measurement, analysis and evaluation (potted version). So why is it that when someone says “Have you tried this” and you ask something in return like “Well, no, but how long will it take and what will be the benefits?” you’ll be lucky to get an answer to the first half of that compound statement, let alone the benefits. “Your students will be happier” is very hard to quantify – but “we reduced our drop-out rate by 30%” is a cold hard fact. (Well, it purports to be. If someone has done their due diligence, it’s a fact.)

There is, of course, a problem. In order to be able to assess the impact of what we’re doing, we have to establish the baseline (how things were before we started), apply our changes, measure the outcomes and then try and determine if what we did had anything to do with the perceived change or whether it was all random noise. This is not helped by the fact that a lot of classes are small, smaller than we need for statistical validity, or that we can’t easily establish cohorts of the right size or consistency. But, being honest, the first problem for many people is that they do not even think of measuring the impact of what they’ve done until after they’ve done it.

So here’s the first of my three slogans.

MIKE: Measurement Is the Key to Everything.

Looking at that model that I keep discussing, I have three separate places to “lose” knowledge in its flow to my students (to reduce the efficacy of flow). The first is in the teaching process itself. If I don’t have the knowledge, I can’t pass it on. If I choose not to share the knowledge, I can’t pass it on. Next, the medium of exchange (that disconnected external transfer from teacher to learner) will make a difference. If I write everything I know in a book and give it to my class, tell them that the final exam is in two months and walk off – I’m in a high-loss environment. So the medium can and does make a difference but it can only facilitate knowledge transfer by minimising the loss or maximising availability of learners to knowledge. It can’t add knowledge. Finally, what the learners themselves do will have a big impact on how the knowledge is processed and assimilated. That’s why, even without curve grading, those Bell curves seem to show up so frequently – in a similar teaching environment, with the same lecturer, individual students still have some variation. We can, of course, vary the peak of the curve but we would expect to see some variation in an otherwise identical environment. A lot of this has to do with the environment that students had before they reached us, which is interesting if only for the fact that this medium of knowledge transfer may now appear to have both memory and temporal aspects – perhaps our dealing with this previous environment, or accepting that such differences exist, in the construction of our transfer medium is as important as the knowledge that we bring to the situation.

Now I can quantify the effort that I put in to my teaching activities, if I’m honest with myself and count time spent actively creating new approaches or materials – and discounting those times I spend in the tea room pontificating about things I never apply. (I don’t think that such sessions have no value, but I hesitate to count them in a genuine measurement of producing new teaching materials unless I am actively mentoring or I run off and do something with that. Even then, I discount the time for each coffee I had. 🙂 ) If I have assessed the student quality or class metric that I want to change, and I have established a baseline on the cohort (somehow), I can come up with an measurement of time spent, or difficulty level to surmount, to implement my new approach and I can then present the effort, and the outcome, along with the environment in order to show other people what I did and how they could do the same thing.

I recently made some changes to a new first year course and I was fortunate in that I achieved a much higher pass rate than usual for the effort that I expended, with excellent process awareness of how to correctly design and finish programming projects on time. Hooray, you might think. Aha – I had only 21 students (it was the first offering and the pipeline was barely filled) and these students had, in the main, correctly self-selected as having programming experience before coming to University. Yes, we had a good result, excellent engagement, and high participation and we achieved it with the standard load model for writing a new course but our environment was not the standard one. Next semester, when I have 130 students from across the range of the intake, I will have an environment where, when I measure how many hours I spent on each activity, I will have much more applicable environment to realistic teaching situations in other Australian Universities.

I’ll be able to assess each student’s early indications of prowess, from their marks in other courses, and compare them to what is achieved in this new course. I can then start to make statements indicating what the benefits of the approach are. But, to do that, I have to think measurement from the moment I start working on the course, keep track of my time, note where I make changes, look for which factors are being affected and, finally, be honest if I can see trends but not significance, an indication of a Bayesian model but not a confirmation. I have to think about quantitative and qualitative assessment mechanisms – I may have to get surveys pre-approved or start designing custom assessment forms. I have to think about how I am going to be able to assess the worth of what I’ve done in the ground-up design of this course weeks before Week 1 – not only for my own benefit, but for communication with others and for possible papers or presentations.

Ultimately, I can give you a warm feeling and tell you that ‘students will love this’ or I can show you the well-written, thoughtful and mature advice on process improvement for timely completion of software projects, well proof-read and easy to read, that I received from the vast majority of the students that I had in my course – after they’d been in the system for less than 12 months. And I could tell you how much effort that took, and the caveats of the environment, and then, with all of those caveats, you might think about how you could do a similar thing in order to achieve a similar result. Or to see if I’m barking mad. That is, after all, what we expect our students to do: assemble evidence, weigh and analyse, complete the evaluation and come to a conclusion. Then act.

Measurement: it really Is the Key to Everything.