The Year(s) of Replication #las17ed L@S 2017

I was at Koli Calling in 2016 and a paper was presented (“Replication in Computing Education Research: Researcher Attitudes and Experiences”) regarding the issue of replicating previous studies. Why replicate previous work? Because we have a larger number of known issues that have emerged in psychology and the medical sciences, where important work has not been able to be replicated. Perhaps the initial analysis was underpowered, perhaps the researchers had terrible bad luck in their sample, and perhaps there were… other things going on. Whatever the reason, we depend upon replication as a validation tool and being unable to replicate work puts up a red flag.


After the paper, I had follow-up discussions with Andrew Petersen, from U Toronto, and we talked about the many problems. If we do choose to replicate studies, which ones do we choose? How do we get the replication result disseminated, given that it’s fundamentally not novel work? When do we stop replicating? What the heck do we do if we invalidate an entire area of knowledge? Andrew suggested a “year of replication” as a starting point but it’s a really big job: how do we start a year of replication studies or commit to doing this as a community?

This issue was raised again at Learning@Scale 2017 by Justin Reich, from MIT, among others. One of the ideas that we discussed as part of that session was that we could start allocating space at the key conferences in the field for replication studies. The final talk as part of L@S was “Learning about Learning at Scale: Methodological Challenges and Recommendations”, which discussed general problems that span many studies and then made recommendations as to how we could make our studies better and reduce the risk of failing future replication. Justin followed up with comments (which he described as a rant but he’s being harsh) about leaving room to make it easier to replicate and being open to this kind of examination of our work: we’re now thinking about making our current studies easier to replicate and better from the outset, but how can we go back and verify all of the older work effectively?

I love the idea of setting aside a few slots in every conference for replication studies. The next challenge is picking the studies but, given each conference has an organising committee, a central theme, and reviewers, perhaps each conference could suggest a set and then the community identify which ones they’re going to have a look at. We want to minimise unnecessary duplication, after all, so some tracking is probably a good idea.

There are several problems to deal with: some political, some scheduling, some scientific, some are just related to how hard it is to read old data formats. None of them are necessarily insurmountable but we have to be professional, transparent and fair in how we manage them. If we’re doing replication studies to improve confidence in the underlying knowledge of the field, we don’t want to damage the community in doing it.

Let me put out a gentle call to action, perhaps for next year, perhaps for the year after. If you’re involved with a conference, why not consider allocating a few slots to replication studies for the key studies in your area, if they haven’t already been replicated? Even the opportunity to have a community discussion about which studies have been effectively replicated will help identify what we can accept as well as showing us what we could fix.

Does your conference have room for a single track, keynote-level session, to devote some time to replication? I’ll propose a Twitter hashtag of #replicationtrack to discuss this and, hey, if we get a single session in one conference out of this, it’s more than we had.

Can we do this? We already have.

How does one actually turn everything I’ve been saying into a course that can be taught? We already have examples of this working, whether in the performance/competency based models found in medical schools around the world or whether in mastery learning based approaches where do not measure anything except whether a student has demonstrated sufficient knowledge or skill to show an appropriate level of mastery.

An absence of grades, or student control over their grades, is not as uncommon as many people think. MIT in the United States give students their entire first semester with no grades more specific than pass or fail. This is a deliberate decision to ease the transition of students who have gone from being leaders at their own schools to the compressed scale of MIT. Why compressed? If we were to assess all school students then we would need a scale that could measure all levels of ability, from ‘not making any progress at school’ to ‘transcendent’. The tertiary entry band is somewhere between ‘passing school studies’ to ‘transcendent’ and, depending upon the college that you enter, can shift higher and higher as your target institution becomes more exclusive. If you look at the MIT entry requirements, they are a little coy for ‘per student’ adjustments, but when the 75th percentile for the SAT components is 800, 790, 790, and 800,800,800 would be perfect, we can see that any arguments on how demotivating simple pass/fail grades must be for excellent students have not just withered, they have caught fire and the ash has blown away. When the target is MIT, it appears the freshmen get their head around a system that is even simpler than Rapaport’s.


Pictured: A highly prestigious University with some of the most stringent entry requirements in the world, which uses no grades in first semester.

Other universities, such as Brown, deliberately allow students to choose how their marks are presented, as they wish to deemphasise the numbers in order to focus on education. It is not a cakewalk to get into Brown, as these figures attest, and yet Brown have made a clear statement that they have changed their grading system in order to change student behaviour – and the world is just going to have to deal with that. It doesn’t seem to be hurting their graduates, from quotes on the website such as “Our 85% admission rate to medical school and 89% admission rate to law school are both far above the national average.

And, returning to medical schools themselves, my own University runs a medical program where the usual guidelines for grading do not hold. The medical school is running on a performance/competency scheme, where students who wish to practise medicine must demonstrate that they are knowledgable, skilful and safe to practice. Medical schools have identified the core problem in my thought experiment where two students could have the opposite set of knowledge or skills and they have come to the same logical conclusion: decide what is important and set up a scheme that works for it.

When I was a solider, I was responsible for much of the Officer Training in my home state for the Reserve. We had any number of things to report on for our candidates, across knowledge and skills, but one of them was “Demonstrate the qualities of an officer” and this single item could fail an otherwise suitable candidate. If a candidate could not be trusted to one day be in command of troops on the battlefield, based on problems we saw in peacetime, then they would be counselled to see if it could be addressed and, if not, let go. (I can assure you that this was not used often and it required a large number of observations and discussion before we would pull that handle. The power of such a thing forced us to be responsible.)

We know that limited scale, mastery-based approaches are not just working in the vocational sector but in allied sectors (such as the military), in the Ivy league (Brown) and in highly prestigious non-Ivy league institutions such as MIT. But we also know of examples such as Harvey Mudd, who proudly state that only seven students since 1955 have earned a 4.0 GPA and have a post on the career blog devoted to “explaining why your GPA is so low” And, be in no doubt, Harvey Mudd is an excellent school, especially for my discipline. I’m not criticising their program, I’ve only heard great things about them, but when you have to put up a page like that? You’re admitting that there’s a problem but you are pushing it on to the student to fix it. But contrast that with Brown, who say to employers “look at our students, not their grades” (at least on the website).

Feedback to the students on their progress is essential. Being able to see what your students are up to is essential for the teacher. Being able to see what your staff and schools are doing is important for the University. Employers want to know who to hire. Which of these is the most important?

The students. It has to be the students. Doesn’t it? (Arguments for the existence of Universities as a self-sustaining bureaucracy system in the comments, if you think that’s a thing you want to do.)

This is not an easy problem but, as we can see, we have pieces of the solution all over the place. Tomorrow, I’m going to put in a place a cornerstone of beautiful assessment that I haven’t seen provided elsewhere or explained in this way. (Then all of you can tell me which papers I should have read to get it from, I can publish the citation, and we can all go forward.)


EduTech AU 2015, Day 2, Higher Ed Leaders, “Innovation + Technology = great change to higher education”, #edutechau

Big session today. We’re starting with Nicholas Negroponte, founder of the MIT Media Lab and the founder of One Laptop Per Child (OLPC), an initiative to create/provide affordable educational devices for children in the developing world. (Nicholas is coming to us via video conference, hooray, 21st Century, so this may or not work well in translation to blogging. Please bear with me if it’s a little disjointed.)

Nicholas would rather be here but he’s bravely working through his first presentation of this type! It’s going to be a presentation with some radical ideas so he’s hoping for conversation and debate. The presentation is broken into five parts:

  1. Learning learning. (Teaching and learning as separate entities.)
  2. What normal market forces will not do. (No real surprise that standard market forces won’t work well here.)
  3. Education without curricula. (Learning comes from many places and situations. Understanding and establishing credibility.)
  4. Where do new ideas come from? (How do we get them, how do we not get in the way.)
  5. Connectivity as a human right. (Is connectivity a human right or a means to rights such as education and healthcare? Human rights are free so that raises a lot of issues.

Nicholas then drilled down in “Learning learning”, starting with a reference to Seymour Papert, and Nicholas reflected on the sadness of the serious accident of Seymour’s health from a personal perspective. Nicholas referred to Papert’s and Minsky’s work on trying to understand how children and machines learned respectively. In 1968, Seymour started thinking about it and on April, 9, 1970, he gave a talk on his thoughts. Seymour realised that thinking about programs gave insight into thinking, relating to the deconstruction and stepwise solution building (algorithmic thinking) that novice programmers, such as children, had to go through.

These points were up on the screen as Nicholas spoke:

  1. Construction versus instruction
  2. Why reinventing the wheel is good
  3. Coding as thinking about thinking

How do we write code? Write it, see if it works, see which behaviours we have that aren’t considered working, change the code (in an informed way, with any luck) and try again. (It’s a little more complicated than that but that’s the core.) We’re now into the area of transferable skills – it appeared that children writing computer programs learned a skill that transferred over into their ability to spell, potentially from the methodical application of debugging techniques.

Nicholas talked about a spelling bee system where you would focus on the 8 out of 10 you got right and ignore the 2 you didn’t get. The ‘debugging’ kids would talk about the ones that they didn’t get right because they were analsysing their mistakes, as a peer group and as individual reflection.

Nicholas then moved on to the failure of market forces. Why does Finland do so well when they don’t have tests, homework and the shortest number of school hours per day and school days per year. One reason? No competition between children. No movement of core resources into the private sector (education as poorly functioning profit machine). Nicholas identified the core difference between the mission and the market, which beautifully summarises my thinking.

The OLPC program started in Cambodia for a variety of reasons, including someone associated with the lab being a friend of the King. OLPC laptops could go into areas where the government wasn’t providing schools for safety reasons, as it needed minesweepers and the like. Nicholas’ son came to Cambodia from Italy to connect up the school to the Internet. What would the normal market not do? Telecoms would come and get cheaper. Power would come and get cheaper. Laptops? Hmm. The software companies were pushing the hardware companies, so they were both caught in a spiral of increasing power consumption for utility. Where was the point where we could build a simple laptop, as a mission of learning, that could have a smaller energy footprint and bring laptops and connectivity to billions of people.

This is one of the reasons why OLPC is a non-profit – you don’t have to sell laptops to support the system, you’re supporting a mission. You didn’t need to sell or push to justify staying in a market, as the production volume was already at a good price. Why did this work well? You can make partnerships that weren’t possible otherwise. It derails the “ah, you need food and shelter first” argument because you can change the “why do we need a laptop” argument to “why do we need education?” at which point education leads to increased societal conditions. Why laptops? Tablets are more consumer-focused than construction-focused. (Certainly true of how I use my tech.)

(When we launched the first of the Digital Technologies MOOCs, the deal we agreed upon with Google was that it wasn’t a profit-making venture at all. It never will be. Neither we nor Google make money from the support of teachers across Australia so we can have all of the same advantages as they mention above: open partnerships, no profit motive, working for the common good as a mission of learning and collegial respect. Highly recommended approach, if someone is paying you enough to make your rent and eat. The secret truth of academia is that they give you money to keep you housed, clothed and fed while you think. )

Nicholas told a story of kids changing from being scared or bored of school to using an approach that brings kids flocking in. A great measure of success.

Now, onto Education without curricula, starting by talking public versus private. This is a sensitive subject for many people. The biggest problem for public education in many cases is the private educational system, dragging out caring educators to a closed system. Remember Finland? There are no public schools and their educational system is astoundingly good. Nicholas’ points were:

  1. Public versus private
  2. Age segregation
  3. Stop testing. (Yay!)

The public sector is losing the imperative of the civic responsibility for education. Nicholas thinks it doesn’t make sense that we still segregate by ages as a hard limit. He thinks we should get away from breaking it into age groups, as it doesn’t clearly reflect where students are at.

Oh, testing. Nicholas correctly labelled the parental complicity in the production of the testing pressure cooker. “You have to get good grades if you’re going to Princeton!” The testing mania is dominating institutions and we do a lot of testing to measure and rank children, rather than determining competency. Oh, so much here. Testing leads to destructive behaviour.

So where do new ideas come from? (A more positive note.) Nicholas is interested in Higher Ed as sources of new ideas. Why does HE exist, especially if we can do things remotely or off campus? What is the role of the Uni in the future? Ha! Apparently, when Nicholas started the MIT media lab, he was accused of starting a sissy lab with artists and soft science… oh dear, that’s about as wrong as someone can get. His use of creatives was seen as soft when, of course, using creative users addressed two issues to drive new ideas: a creative approach to thinking and consulting with the people who used the technology. Who really invented photography? Photographers. Three points from this section.

  1. Children: our most precious natural resource
  2. Incrementalism is the enemy of creativity
  3. Brain drain

On the brain drain, we lose many, many students to other places. Uni are a place to solve huge problems rather than small, profit-oriented problems. The entrepreneurial focus leads to small problem solution, which is sucking a lot of big thinking out of the system. The app model is leading to a human resource deficit because the start-up phenomenon is ripping away some of our best problem solvers.

Finally, to connectivity as a human right. This is something that Nicholas is very, very passionate about. Not content. Not laptops. Being connected.  Learning, education, and access to these, from early in life to the end of life – connectivity is the end of isolation. Isolation comes in many forms and can be physical, geographical and social. Here are Nicholas’ points:

  1. The end of isolation.
  2. Nationalism is a disease (oh, so much yes.) Nations are the wrong taxonomy for the world.
  3. Fried eggs and omelettes.

Fried eggs and omelettes? In general, the world had crisp boundaries, yolk versus white. At work/at home. At school/not at school. We are moving to a more blended, less dichotomous approach because we are mixing our lives together. This is both bad (you’re getting work in my homelife) and good (I’m getting learning in my day).

Can we drop kids into a reading environment and hope that they’ll learn to read? Reading is only 3,500 years old, versus our language skills, so it has to be learned. But do we have to do it the way that we did it? Hmm. Interesting questions. This is where the tablets were dropped into illiterate villages without any support. (Does this require a seed autodidact in the group? There’s a lot to unpack it.) Nicholas says he made a huge mistake in naming the village in Ethiopia which has corrupted the experiment but at least the kids are getting to give press conferences!

Another massive amount of interesting information – sadly, no question time!


SIGCSE, Keynote #2, Hal Abelson, “The midwife doesn’t get to keep the baby.”

Well, another fantastic keynote and, for the record, that’s not the real title. The title of the talk was From Computational Thinking to Computational Values. For those who don’t know who Hal Abelson is, he’s a Professor of EE/CS at MIT who has made staggering contributions to pedagogy and the teaching of Computer Science over the years. He’s been involved with the first implementations of Logo, changed the way we think about using computer languages, has been a cornerstone of the Free Software Movement (including the Foundation), led the charge of the OpenCourseWare (OCW) at MIT, published many things that other people would have been scared to publish and, basically, has spent a long time trying to make the world a better place.

It went without saying that, today, we were in for some inspiration and, no doubt, some sort of call to arms. We weren’t disappointed. What follows is as accurate a record as I could make, typing furiously. I took a vast quantity of notes over what was a really interesting talk and I’ll try to get the main points down here. Any mistakes are mine and I have tried to represent the talk without editorialising, although I have adjusted some of the phrasing slightly in places, so the words are, pretty much, Professor Abelsons’s.

Professor Abelson started from a basic introduction of Computational Thinking (CT) but quickly moved on to how he thought that we’d not quite captured it properly in modern practice: it’s how we look in this digital world and see it as a source of empowerment for everybody, as a life changing view. Not just CT, but computational values.

What do we mean? We’re not only talking about cool ideas but that these ideas should be empowering and people should be able to exercise great things and have an impact on the world.

He then went on to talk about Google’s Ngram viewer, which allows you to search all of the books that Google has scanned in and find patterns. You can use this to see how certain terms, ideas and names come and go over time. What’s interesting here is that (1) ascent to and descent from fame appears to be getting faster and (2) you can visualise all of this and get an idea of the half-life of fame (which was nearly the title of this post).

Abelson describes this as a generative platformone which can be used for things that were not thought of it when it was built, one we can build upon ourselves and change over time. Generating new things for an unseen future. (Paper reference here was Nature, with a covering article from another magazine entitled “Researchers Aim to chart intellectual trends in Arxiv”)

Then the talk took a turn. Professor Abelson took us back, 8 years ago, when Duke’s “Give everyone an iPod” project had every student (eventually) with a free iPod and encouraged them to record, share and mix-up what they were working with.

Enter the Intellectual Property Lawyer. Do the students have permission to share the lecturer-created creative elements of the lectures?

Professor Abelson’s point is that we are booming more concerned with locking up our content into proprietary Content Management Systems (CMS) and this risks turning the academy into a marketplace for packaged ideas and content, rather than a place of open enquiry and academic freedom. This was the main theme of the talk and we’ve got a lot of ground left to cover here! This talk was for those who loved computational values, rather than property creation.

We visited the early, ham-fisted attempts to grant limited licences for simple activities like recording lectures and the immediately farcical notion that I could take notes of a lecture and be in breach of copyright if I then discussed it with a classmate who didn’t attend. Ngrams shows what happens when you have a system where you can do what you like with the data – what if the person holding that data for you, which you created, starts telling you what to do? Where does this leave our Universities?

Are we producing education or property? Professor Abelson sees this as a battle for the soul of the Universities. We should be generative.

We can take computational actions, actions that we will take to reinforce the sense that we have that people ought to be able to relish the power that they get from our computational thinking and computational ideas. This includes providing open courseware (like MIT’s OCW and Stanford’s AI) and open access to research, especially (but not only) when funded by the public purse.

As a teaser, at this point, Abelson introduced MITx, an online intensive learning system that opens up on MONDAY. No other real details – put it in your calendar to check out on Monday! MIT want their material and their content engines to be open source and generative – that word again! Put it into your own context or framework and do great things!

The companion visions to all of this are this:

  1. Great learning institutions provide universal access to course content. (OpenCourseWare)
  2. Great research institutions provide universal access to their collective intellectual resources.(DSpace)

What are the two reasons that we should all support these open initiatives? Why should we fill in the moat and open the drawbridge?

  1. Without initiatives to maintain them, we risk marginalising our academic values and stressing our university communities.
  2. To keep a seat at the table in decisions about the disposition of knowledge in the information age.

Abelson introduced an interesting report, “Who Owns Academic Work? Battling for Control of Intellectual Property”, which discusses the conflation of property and academic rights.

Basically, scientific literature has become property. We, academia, produce it and then give away our rights to journal publishers, who give us limited rights in exchange on a personal level and then hold onto it forever. Neither our institution nor the public has any right to this material anymore. We looked at some examples of rights. Sign up to certain publishers and, from that point on, you can use only up to 250 words of any of the transferred publications in a new work. The number of publishers is shrinking and the cost of subscription is rising.

Professor Abelson asked how it is that, in this sphere alone, the midwife gets to keep the baby? We all have to publish if we act individually, as promotions and tenure depend upon publication in prominent journals – but that there was hope (and here he referred to the Mathematical boycott of the Elsevier publishing group). HR 3699 (the Research Works Act) could have challenged any federal law that mandated open access on federally funded research. Lobbied for by the journal publishing group, it lost support, firstly from Elsevier, and then from the two members of Congress who proposed it

Even those institutions that have instituted an open access policy are finding it hard – some publishers have made specific amendments to the clause that allows pre-print drafts to be display locally to say “except where someone has an institutionally mandated open access policy”.

BUT. HR3699 has gone away for now. Abelson’s message is that there is hope!

We have allowed a lot of walled gardens to spring up. Places where data is curated and applications made available, but only under the permission of the gardener. Despite our libraries paying up to hundreds of thousands of dollars for access to the on-line journal stores, we are severely limited in what we can do with them. Your library cannot search it, index it, scrape it, or many other things. You can, of course, buy a service that provides some of these possibilities from the publisher. A walled garden is not a generative environment.

Jonathan Zittrain, 2008, listed two important generative technologies: the internet and the PC, because you didn’t need anyone’s permission to link or to run software. In Technology Review, now, Zittrain thinks that the PC is dead because of the number of walled gardens that have sprung up.

In Professor Abelson’s words:

Network Effects
lead to
Monopoly Positions
lead to 
Concentration of Channels
lead to
Decline of Generativity.
 What about tomorrow? Will our students have the same tinkering possibilities that we had? Will any of our old open software still run?  Will mobile computing be tinkerable? Open source allows for small tinkering steps, and reduces our reliance on monolithic, approved, releases.
The talk then concluded with some more of Professor Abelson’s words, which I reproduce here because they are far better than mine.
We have the spark of inspiration about how one should relate to their information environment and the belief that that kind of inspiration, power and generativity should be available to everybody.

These beliefs are powerful and have powerful enemies. Draw on your own inspiration and power to make sure that what inspired us is going to be available to our students.