The Year(s) of Replication #las17ed L@S 2017Posted: April 22, 2017 Filed under: Education, Opinion | Tags: community, education, l@s, las17ed, learning, learning@scale, mit, replication, science, teaching, testing 6 Comments
I was at Koli Calling in 2016 and a paper was presented (“Replication in Computing Education Research: Researcher Attitudes and Experiences”) regarding the issue of replicating previous studies. Why replicate previous work? Because we have a larger number of known issues that have emerged in psychology and the medical sciences, where important work has not been able to be replicated. Perhaps the initial analysis was underpowered, perhaps the researchers had terrible bad luck in their sample, and perhaps there were… other things going on. Whatever the reason, we depend upon replication as a validation tool and being unable to replicate work puts up a red flag.
After the paper, I had follow-up discussions with Andrew Petersen, from U Toronto, and we talked about the many problems. If we do choose to replicate studies, which ones do we choose? How do we get the replication result disseminated, given that it’s fundamentally not novel work? When do we stop replicating? What the heck do we do if we invalidate an entire area of knowledge? Andrew suggested a “year of replication” as a starting point but it’s a really big job: how do we start a year of replication studies or commit to doing this as a community?
This issue was raised again at Learning@Scale 2017 by Justin Reich, from MIT, among others. One of the ideas that we discussed as part of that session was that we could start allocating space at the key conferences in the field for replication studies. The final talk as part of L@S was “Learning about Learning at Scale: Methodological Challenges and Recommendations”, which discussed general problems that span many studies and then made recommendations as to how we could make our studies better and reduce the risk of failing future replication. Justin followed up with comments (which he described as a rant but he’s being harsh) about leaving room to make it easier to replicate and being open to this kind of examination of our work: we’re now thinking about making our current studies easier to replicate and better from the outset, but how can we go back and verify all of the older work effectively?
I love the idea of setting aside a few slots in every conference for replication studies. The next challenge is picking the studies but, given each conference has an organising committee, a central theme, and reviewers, perhaps each conference could suggest a set and then the community identify which ones they’re going to have a look at. We want to minimise unnecessary duplication, after all, so some tracking is probably a good idea.
There are several problems to deal with: some political, some scheduling, some scientific, some are just related to how hard it is to read old data formats. None of them are necessarily insurmountable but we have to be professional, transparent and fair in how we manage them. If we’re doing replication studies to improve confidence in the underlying knowledge of the field, we don’t want to damage the community in doing it.
Let me put out a gentle call to action, perhaps for next year, perhaps for the year after. If you’re involved with a conference, why not consider allocating a few slots to replication studies for the key studies in your area, if they haven’t already been replicated? Even the opportunity to have a community discussion about which studies have been effectively replicated will help identify what we can accept as well as showing us what we could fix.
Does your conference have room for a single track, keynote-level session, to devote some time to replication? I’ll propose a Twitter hashtag of #replicationtrack to discuss this and, hey, if we get a single session in one conference out of this, it’s more than we had.
5 Things: ScientistsPosted: September 17, 2014 Filed under: Education, Opinion | Tags: advocacy, authenticity, Biological scientists, blogging, community, education, einstein, five things, Google image search, learninge, newton, research, science, scientists, stereotype, stereotype threat, teaching, thinking, white coats Leave a comment
Another 5-pointer, inspired by a post I read about the stereotypes of scientists. (I know there are just as many about other professions but scientist is one of my current ones.)
- We’re not all “bushy-haired” confused old white dudes.
It’s amazing that pictures of 19th Century scientists and Einstein have had such an influence on how people portray scientists. This link shows you how academics (researchers in general but a lot of scientists are in here) are shown to children. I wouldn’t have as much of a problem with this if it wasn’t reinforcing a really negative stereotype about the potential uselessness of science (Professors who are not connected to the real world and who do foolish things) and the demography (it’s almost all men and white ones at that) which are more than likely having a significant impact on how kids feel about going into science.
It’s getting better, as we can see from a Google image search for scientists, which shows a very obvious “odd man out”, but that image search actually throws up our next problem. Can you see what it is?
- We don’t all wear white coats!
So we may have accepted that there is demographic diversity in science (but it still has to make it through to kid’s books) but that whole white coat thing is reinforced way too frequently. Those white coats are not a uniform, they’re protective clothing. When I was a winemaker, I wore heavy duty dark-coloured cotton clothing for work because I was expecting to get sprayed with wine, cleaning products and water on a regular basis. (Winemaking is like training an alcoholic elephant with a mean sense of humour.) When I was in the lab, if I was handling certain chemicals, I threw on a white coat as part of my protective gear but also to stop it getting on my clothes, because it would permanently stain or bleach them. Now I’m a computer scientist, I’ve hung up my white coat.
Biological scientists, scientists who work with chemicals or pharmaceuticals – any scientists who work in labs – will wear white coats. Everyone else (and there’s a lot of them) tend not to. Think of it like surgical scrubs – if your GP showed up wearing them in her office then you’d think “what?” and you’d be right.
- Science can be a job, a profession, a calling and a hobby – but this varies from person to person.
There’s the perception of scientist as a job so all-consuming that it robs scientists of the ability to interact with ‘normal’ people, hence stereotypes like the absent-minded Professor or the inhuman, toxic personality of the Cold Scientific Genius. Let’s tear that apart a bit because the vast majority of people in science are just not like that.
Some jobs can only be done when you are at work. You do the work, in the work environment, then you go home and you do something else. Some jobs can be taken home. The amount of work that you do on your job, outside of your actual required working time – including overtime, is usually an indicator of how much you find it interesting. I didn’t have the facilities to make wine at home but I read a lot about it and tasted a lot of wine as part of my training and my job. (See how much cooler it sounds to say that you are ‘tasting wine’ rather than ‘I drink a lot’?) Some mechanics leave work and relax. Some work on stock cars. It doesn’t have to be any particular kind of job because people all have different interests and different hobbies, which will affect how they separate work and leisure – or blend them.
Some scientists leave work and don’t do any thinking on things after hours. Some can think on things but not do anything because they don’t have the facilities at home. (The Large Hadron Collider cost close to USD 7 Billion, so no-one has one in their shed.) Some can think and do work at home, including Mathematicians, Computer Scientists, Engineers, Physicists, Chemists (to an extent) and others who will no doubt show up angrily in the comments. Yes, when I’m consumed with a problem, I’m thinking hard and I’m away with the pixies – but that’s because, as a Computer Scientist, I can build an entire universe to work with on my laptop and then test out interesting theories and approaches. But I have many other hobbies and, as anyone who has worked with me on art knows, I can go as deeply down the rabbit hole on selecting typefaces or colours.
Everyone can appear absent-minded when they’re thinking about something deeply. Scientists are generally employed to think deeply about things but it’s rare that they stay in that state permanently. There are, of course, some exceptions which leads me to…
- Not every scientist is some sort of genius.
Sorry, scientific community, but we all know it’s true. You have to be well-prepared, dedicated and relatively mentally agile to get a PhD but you don’t have to be crazy smart. I raise this because, all too often, I see people backing away from science and scientific books because “they wouldn’t understand it” or “they’re not smart enough for it”. Richard Feynman, an actual genius and great physicist, used to say that if he couldn’t explain it to Freshman at College then the scientific community didn’t understand it well enough. Think about that – he’s basically saying that he expects to be able to explain every well-understood scientific principle to kids fresh out of school.
The genius stereotype is a not just a problem because it prevents people coming into the field but because it puts so much demand on people already in the field. You could probably name three physicists, at a push, and you’d be talking about some of the ground-shaking members of the field. Involved in work leading up those discoveries, and beyond, are hundreds of thousands of scientists, going about their jobs, doing things that are valuable, interesting and useful, but perhaps not earth-shattering. Do you expect every soldier to be a general? Every bank clerk to become the general manager? Not every scientist will visibly change the world, although many (if not most) will make contributions that build together to change the world.
Sir Isaac Newton, another famous physicist, referred to the words of Bernard of Chartres when he famously wrote:
“If I have seen further it is by standing on the sholders [sic] of Giants”
making the point even more clearly by referring to a previous person’s great statement to then make it himself! But there’s one thing about standing on the shoulders of giants…
- There’s often a lot of wrong to get to right.
Science is evidence-based, which means that it’s what you observe occurring that validates your theories and allows you to develop further ideas about how things work. The problem is that you start from a position of not knowing much, make some suggestions, see if they work, find out where they don’t and then fix up your ideas. This has one difficult side-effect for non-scientists in that scientists can very rarely state certainty (because there may be something that they just haven’t seen yet) and they can’t prove a negative, as you just can’t say something won’t happen because it hasn’t happened yet. (Absence of evidence is not evidence of absence.) This can be perceived as weakness but it’s one of the great strengths of science. We work with evidence that contradicts our theories to develop our theories and extend our understanding. Some things happen rarely and under only very specific circumstances. The Large Hadron Collider was built to find evidence to confirm a theory and, because the correct tool was built, physicists now better understand how our universe works. This is a Good Thing as the last thing we want do is void the warranty through incorrect usage.
The more complicated the problem, the more likelihood that it will take some time to get it right. We’re very certain about gravity, in most practical senses, and we’re also very confident about evolution. And climate change, for that matter, which will no doubt get me some hate on the comments but the scientific consensus is settled. It’s happening. Can we say absolutely for certain? No, because we’re scientists. Again – strength, not weakness.
When someone gets it wrong deliberately, and that sadly does happen occasionally, we take it very seriously because that whole “standing on shoulders of giants” is so key to our approach. A disingenuous scientist, like Andrew Wakefield and his shamefully bad and manipulated study on vaccination that has caused so much damage, will take a while to be detected and then we have to deal with the repercussions. The good news is that most of the time we find these people and limit their impact. The bad news is that this can be spun in many ways, especially by compromised scientists, and humans can be swayed by argument rather than fact quite easily.
The take away from this is that admitting that we need to review a model is something you should regard in the same light as your plane being delayed because of a technical issue. You’d rather we fixed it, immediately and openly, than tried to fly on something we knew might fail.
Proud to be a #PreciousPetal, built on a strong #STEM, @PennyWrites @SenatorMilne @adambandtPosted: September 11, 2014 Filed under: Education, Opinion | Tags: advocacy, Australia, authenticity, blogging, community, education, ethics, higher education, Ian Macfarlane, Macfarlane, Minister for Industry, Minister for Science, politics, precious petal, science, thinking Leave a comment
I am proud to be a Precious Petal. Let me explain why I think we should reclaim this term for ourselves.
Australia, apparently, does not have a need for dedicated Science Minister, for the first time since the 1930s. Instead, it is a subordinate portfolio for our Minister for Industry, the Hon Ian Macfarlane, MP. Today, he was quoted in the Guardian, hitting out at “precious petals in the science industry” who are criticising the lack of a dedicated Science Minister. Macfarlane, whose Industry portfolio includes Energy, Skills and Science went on to say:
“I’m just not going to accept that crap,” he said. “It really does annoy me. There’s no one more passionate about science than me, I’m the son and the grandson of a scientist. I hear this whinge constantly from the precious petals in the science industry.”
So I’m not putting words in his mouth – that’s a pretty directed attack on the sector that happens to underpin Energy and Industry because, while Macfarlane’s genetic advantage in his commitment to science may or not be scientifically valid, the fact of the matter is that science, and innovation in science, have created pretty much all of what is referred to as industry in Australia. I’m not so one-eyed as to say that science is everything, because I recognise and respect the role of the arts and humanities in a well-constructed and balanced society, but if we’re going to talk about everything after the Industrial (there’s that word again) Revolution in terms of production industries – take away the science and we’re not far away poking things with sticks to work out which of the four elements (fire, air, earth, water) it belongs to. Scientists of today stand on a tradition of thousands of years of accumulated knowledge that has survived many, many regimes and political systems. We tell people what the world is like, rather than what people want it to be, and that often puts us at odds with politicians, for some reason. (I feel for the ethicists and philosophers who have to do the same thing but can’t get industry implementation partnerships as easily and are thus, unfairly, regularly accused of not being ‘useful’ enough.)
I had the opportunity to be addressed by the Minister at Science Meets Parliament where, like something out of a David Williamson play, the genial ageing bloke stood up and, in real Strine, declaimed “No Minister for Science? I’m your Minister for Science!” as if this was enough for a room full of people who were dedicated to real evidence. But he obviously thought it was enough as he threw a few bones to the crowd. On the back of the cuts to CSIRO and many other useful scientific endeavours, these words ring even more hollow than they did at the time.
But rather than take offence at the Minister’s more recent deliberately inflammatory and pejorative words, let me take them and illustrate his own lack of grasp of his portfolio.
My discipline falls into STEM – Science, Technology, Engineering and Mathematics – and I am scientist in that field. Personally, I like to add an A for Arts, as I am rather cross-disciplinary, and make it STEAM, because that conveys the amazing potential and energy in the area when we integrate across the disciplines. So, if Science is a flower, then we have a strong STEM in Australia, although it is currently under threat from a number of initiatives put in place by this very government.
But what of petals? If the Minister knew much botany, he’d know that petals are modified leaves that protect parts of the flower, attract or deliberately drive away certain pollinators, building relationships with their pollinating community to build a strong ecosystem. When flowers have no petals, they are subject to the whim on the winds for pollination and this means that you have to be very wasteful in your resources to try and get to any other plants. When the petals are strong and well-defined, you can draw in the assistance of other creatures to help you use your resources more wisely and achieve the goals of the flower – to produce more flowers over time.
At a time when bee colony collapse is threatening agriculture across the globe, you would think that a Minister of Industry (and Science) would have actually bothered to pick up some of the facts on this, very basic, role of a mechanism that he is using to deride and, attempt to, humiliate a community for having the audacity to complain about a bad decision. Scientists have been speaking truth to power since the beginning, Minister, and we’re not going to stop now.
If the Minister understood his portfolio, then he would realise that calling Australia’s scientific community “precious petals” is actually a reflection of their vital role in making science work for all Australians and the world. It is through these petals, protecting and guiding the resources in their area, that we can take the promise of STEM and share it with the world.
But let’s not pretend that’s what he meant. Much like the staggering Uncle at a Williamson Wedding, these words were meant to sting and diminish – to make us appear hysterical and, somehow, less valid. In this anachronistic, and ignorant, attack, we have never seen a better argument as to why Australia should have a dedicated Science Minister, who actually understands science.
I’m proud to be a Precious Petal, Minister.
Dr Falkner Goes to Canberra Day 2, “Q and A”, (#smp2014 #AdelEd @foreignoffice)Posted: March 18, 2014 Filed under: Education | Tags: Chief Scientific Advisor, collaboration, education, EU, european union, foreign office, higher education, research, science, Science Meets Parliament, smp2014 Leave a comment
The last formal event was a question and answer session with Professor Robin Grimes, the Chief Scientific Advisor for the UK Foreign and Commonwealth Office (@foreignoffice). I’ll recover from Question Time and talk about it later. The talk appears to have a secondary title of “The Role of the CSA Network, CSAs in SAGE, the CSA in the FCO & SIN”. Professor Grimes started by talking about the longstanding research collaboration between the UK and AUS. Apparently, it’s a unique relationship (in the positive sense), according to William Hague. Once again, we come back to explaining things to non-scientists or other scientists.
There are apparently a number of Chief Scientists who belong to the CSA Network, SAGE – Scientific Advisory Group for Emergencies – and the Science and Innovation Network (SIN). (It’s all a bit Quartermass really.) And here’s a picture that the speaker refers to as a rogue’s gallery. We then saw a patchwork quilt that shows how the UK Government Science Advisory Structure, which basically says that they work through permanent secretaries and ministers and other offices – imagine a patchwork quilt representation of the wars in the Netherlands as interpreted by Mondrian in pastoral shades and you have this diagram. There is also another complex diagram that shows that laboratories are many and advisors scale.
Did you know that there is a UK National Risk Register? Well, there is, and there’s a diagram with blue blobs and type I can’t see from the back of the room to talk about it. (Someone did ask why they couldn’t read it and the speaker joked that it was restricted. More seriously, things are rated on their relative likelihood and relative impact.
The UK CSAs and FCO CS are all about communication, mostly by acronym apparently. (I kid.) Also, Stanley Baldwin’s wife could rock a hat. More seriously, fracking is an example of poor communication. Scientific concerns (methane release, seismic events and loss of aquifer integrity) are not meeting the community concerns of general opposition to oil and gas and the NIMBY approach. The speaker also mentioned the L’Aquila incident, where scientists were convicted of a crime for making an incorrect estimation of the likelihood of a seismic event. What does this mean for scientific advice generally? (Hint: don’t give scientific advice in Italy.) Scientists should feel free to express their view and understanding conceding risks, their mitigation and management, freely to the government. If actions discourage scientists from coming forward, then it;s highly undesirable. (UK is common law so the first legal case will be really, really interesting in this regard.)
What is the role of the CSA in emergencies? This is where SAGE comes in. They are “responsible for coordinating and peer reviewing, as far as possible, scientific and technical advice to inform decision-making”. This is chaired by the GCSA, who report to COBRA (seriously! It’s the Cabinet Office Briefing Room A) and includes CSAs, sector experts and independent scientists. So swine flu, volcanic ash cloud, Fukushima and the Ash die-back – put up the SAGE signal!
What’s happened with SAGE intervention? Better relationships with science diplomacy. Also, when the media goes well, there is a lot of good news to be had.
The Foreign Office gets science-based advice which relate to security, prosperity and consular – the three priorities of the Foreign Office. It seems that everyone has more science than us. There are networks for Science Networks, Science Evidence and Scientific Leadership, but the Foreign Office is a science-using rather than a science-producing department. There is no dedicated local R&D, scientists and engineer cadre or a departmental science advisory committee.
The Science and Innovation Network (SIN) has two parent departments, Foreign Office (FCO) and Department for Business, Innovation and Skills (BIS, hoho). And we saw a slide with a lot of acronyms. This is the equivalent of the parents being Department of Industry and DFAT for our system. 90 people over 28 countries and territories, across 46 cities. There’s even one here (where here is Melbourne and Canberra). (So we support UK scientists coming out to do cool science here. Which is good. If only we had a Minister for Science, eh?) Apparently they produce newsletters and all sorts of tasty things.
They even talk to the EU (relatively often) and travels to the EU quite frequently in an attempt to make the relationships work through the EU and bi-laterally. There aren’t as many Science and Innovation officers in the EU as they can deal directly with the EU. There are also apparently a lot of student opportunities (sound of ears pricking up) but it’s for UK students coming to us. There are also opportunities for UK-origin scientists to either work back in the UK or for them to bring out UK academics that they know. (Paging Martin White!)
There is a Newton Fund (being developed at the moment), a science-based aid program for countries that are eligible for official development assistance (ODA) and this could be a bi-lateral UK-AUS collaboration.
Well, that’s it for the formal program. There’s going to be some wrap-up and then drinks with Adam Bandt, Deputy Leader of the Australian Greens, Hope you’ve enjoyed this!
Dr Falkner Goes to Canberra Day 1 “Getting your science out of the lab” (#smp2014 #AdelED @rodl @willozap)Posted: March 17, 2014 Filed under: Education | Tags: 60 second pitch, communicating science, communication, education, higher education, schmooze, science, Science Meets Parliament, smp2014 Leave a comment
Ok, the play is over and it was interesting but effectively uncaptureable in this format – you can see it in the Tweet stream. Sorry about that. The speakers for this session Dr Rod Lambert and Dr Will Grant, both of the ANU Centre for the Public Awareness of Science. Here’s a chance to practice a 60 second pitch of your science – which will be held after the convenience break.
It’s been an intense morning so the banter, which would normally be quite engaging, appears to be losing some of the people in the audience from looking around. Hopefully they’ll settle in shortly.
A question from the floor – there’s some anxiety about the purpose of the meeting, how to approach it, how to build for it and how to build on it afterwards. Answer: all communication is personal. What are you trying to say? Say it to the person opposite you as if they are a person – because they are. Remember that the politician is happy to meet you – let’s start from there and build some common ground. Stick to your own persona and be yourself. You can tailor your personality but don’t go overboard. The meeting may not go to the details you want, but don’t overwhelm with caveats and detail. Listen to the other person, hear their values and motivations and, once again, tailor your message.
Question from the floor – scientists are trained to respond in a particular way where we are very rarely certain but we are actually quite certain enough for most intentions out in the community. How much do you present the uncertainty (without misrepresenting it). Politicians don’t act in certainty and have to make risk calculations before they take action. Talk enough but know when to stop.
You can’t offer a doom and gloom – you need to offer a solution and a way out, something that the politician can do. Be simple in discussions: “My research can do X, then possibly label it.” If you get a blank look, back off, shift down, go forward. You are aiming to communicate, not impress with your intellect. Remember you’re representing someone and you can fall back onto their agendas!
I realise many of my friends don’t necessarily agree with this but the common message is that you have to be good at the show, as well as good at the science, because without communications it doesn’t get communicated. It makes me a bit sad to agree with this because I know some wonderful people who aren’t as great in front of an audience – but this means you need a screaming front person who can help you with it to get the ‘right’ people listening.
And, yes, I’m for hire. (You’ll need to speak to my manager.)