Bad Summaries Ruin Good Reports: Generation Why?

A media release came around on Friday from Universities Australia called “Generation WhY? (sic) Students question point of science and maths“. You can read the media release, key findings and the associated report here. The key findings, for students who are both STEM and non-STEM, are published with a series of pull quotes and explanations underneath them. For my own purposes, I’ve removed those because I want you to read the key findings in the raw:

1. More than 40% of students surveyed did not feel encouraged to do well in maths and science by their teachers at high school
2. 1 in 3 students were influenced by past teachers in their university choices
3. 1 in 5 STEM students somewhat or totally engage in the stereotype that science is for nerds.
4. Students sometimes felt (that) STEM subjects do not align with other interests and abilities.
5. Some students interviewed saw no positive value from pursuing STEM as a career.
6. An inability to understand or work with the precise black-and-white nature of science, as opposed to less structured processes, turned some students away.

The report itself is 144 pages long but from page 88 it’s an appendix containing the survey so it’s not too long a read. However, those 6 statements above are, well, in the politest way possible, not very precise. Finding 2., for example, has accompanying text that implies the influence was negative – that 1 in 3 students were discouraged, rather than influenced. Finding 3 is interesting but how many non-STEM students feel the same way? Finding 4 – sometimes? Let’s look at the questions to get an idea of how the survey was framed. The initial questions are all basic demography, then we get to the meat.

Question 19. As a person are you primarily?

• More socially outgoing and like being the centre of attention
• More a quiet and private person and like being with your own thoughts
• Not sure/can’t answer

Urm. I’m socially outgoing but I like time alone with my thoughts. I’m sure of that, however. But this is a quibble. Not many people will have a problem with this. Let’s look at another one.

Question 21. As a person do you primarily?

• Go with your gut instincts
• Focus on cold hard facts
• Not sure/Can’t answer

Urm, again. COLD HARD FACTS. M’lud, I think that we’re leading the witness a tad. How about “Go with your instincts/Focus on the facts”? (Still lots of room for improvement)

You can read the rest of the survey yourself – because I don’t want you all to die of boredom. First thing is that, yes, of course, survey design is hard and I’m sure that a lot of thought went into this survey. However, the press release that came out from this survey makes some claims that, if true, mean that we in the higher edu sector are pretty much stuffed in some ways, because we just won’t get the students here to work with in the first place. Once a student gets into STEM, I can work with them. If, as the survey suggests, I’m losing 33% to teacher discouragement, or 40% to not doing well, or 20% to the nerd factor, I’ve lost a vast number of potential students.

Reading the survey, rather than the keypoints, is far more illuminating. It turns out that teacher influence can be either way, which should have been obvious in the summary. It paints teachers in a much fairer manner. That whole ‘science is for nerds’ is in the middle of a question with lots of opinion options and a 5 point rating scale for agreement. So 20% of STEM students ticked the Totally or Somewhat agree box.

Hang on. That means that 80% of the people in STEM either can’t answer or don’t think it’s for nerds. Page 69 of the report talks on this. I quote: “A higher proportion of STEM respondents somewhat agreed with the statement science is for nerds than did non-STEM respondents.”

They then show the results table. 1364 students in total, 730 non-STEM, 634 STEM. 96 of non-STEM thought it was for nerds, 124 of STEM thought it was for nerds. All other results were disagreers. They’ve already removed the can’t answer people from the survey. That’s 13% of outside STEM people and 19.6% of STEM. Now all of these students are currently enrolled, at University, so the people who are more likely to think science is for nerds are already inside our borders. So, the actual finding is:

“Around 1 in 10 students outside of STEM have a negative image of science as being for nerds, and the number increases slightly to just under 1 in 5 for students inside STEM. Overall, roughly 1 in 6 first-year students surveyed have a perception of science as nerdy.”

That’s surprisingly positive to me. I’d always thought that everyone thought we were enormous dorks. Hooray! Checking the figures, only 5% of STEM students totally agree anyway, compared with 3% of non-STEM, but we have a lot more ‘somewhat agrees’ which really drives the numbers up in STEM.

Here’s the quote that was underneath the 1 in 5 figure in finding 3: “Also if you see scientists on the news like, there’s kind of a stereotype that you will see… Like kind of wearing glasses… They never dress well.” That seems pretty damning. Not only do people think we’re nerds, they took the time to write this down.

But that quote doesn’t come from the survey. That pull quote is not from the same source as the survey data, it’s an anonymous student comment from the Phase 1 pre-survey focus group. In fact, there is no text box associated with that question (Question 80) – Question 81 is a question with a text box, but it’s for comments about the survey itself. Associating that quote with that finding makes a very strong implied linkage that is very. simply. not. there. The initial focus group at University of Sydney was composed of 8 people, a 5/3 male/female split, all first-year, with five B.Sc and three B.A. students. What they admitted that they felt about stereotyping was used to build the survey question at the end. But putting their pre-survey thoughts together with a post-survey result is something that, well, ok, maybe it’s done all the time, but I wouldn’t do it myself.

Those two entities have no linkage – unless it is to say “Hey, the focus group thought everyone would think that science was for nerds but they turned out to be wrong – it’s less than 20% on average and we’re harder on ourselves in STEM, about being cool, than other people think we should be. Woo!” because that recognises the data origin and what the result means. The way that it is presented in the key findings is misleading.

Finding 4 is a curious one (Students sometimes felt (that) STEM subjects do not align with other interests and abilities.) because there is a question, Q50, that asks about why you chose a particular degree. However, the report does not clearly show the detail of the responses and the question just lists ‘Best fit for my interests and abilities’ as one of the options for “What are the reasons for your choice of University degree/course/program”. Searching for the words “interests” or “abilities” in the text brings up some earlier quotes and I must be missing something because I couldn’t find anything to support finding 4, beyond a brief quote from the pre-focus group again. The word ‘align’ doesn’t occur in the report. I’ve read all the questions and can’t see where that finding could be derived. I must be missing something because I can’t find a single solid point in the report, or a summary, that supports this key finding. So, dear reader, if you can find it, please help me out and show me where it is! (I’m a bit tired, so forgive me if I’ve missed the obvious.)

I can’t help but feel that this media release, focusing on negative interpretation and using contextualising quotes that reinforce that interpretation, is doing a disservice to the interesting data contained within the report. Check it out for yourself to see how else things have been reported one way in the actual report and then projected out through the media release. If nothing else, it’s a teaching example in itself of how you can present data accurately but in a way that will very definitely channel someone’s interpretation – especially if they don’t bother to read the original article. If you read the report, you can see that the writers are concerned about the statistical validity because only 12% of their target group responded.

It’s a reminder that all the work you put into your survey design and data analysis process is nothing if that message is lost or adulterated in the search for an easy message. The message matters more than the medium. Once again, the medium is important, but the message is paramount.

Finally, it’s a reminder that we always must read the primary source, to at least calibrate the secondary and tertiary reports.