Designing a MOOC: how far did it reach? #csed

Mark Guzdial posted over on his blog on “Moving Beyond MOOCS: Could we move to understanding learning and teaching?” and discusses aspects (that still linger) of MOOC hype. (I’ve spoken about MOOCs done badly before, as well as recording the thoughts of people like Hugh Davis from Southampton.) One of Mark’s paragraphs reads:

“The value of being in the front row of a class is that you talk with the teacher.  Getting physically closer to the lecturer doesn’t improve learning.  Engagement improves learning.  A MOOC puts everyone at the back of the class, listening only and doing the homework”

My reply to this was:

“You can probably guess that I have two responses here, the first is that the front row is not available to many in the real world in the first place, with the second being that, for far too many people, any seat in the classroom is better than none.

But I am involved in a, for us, large MOOC so my responses have to be regarded in that light. Thanks for the post!”

Mark, of course, called my bluff and responded with:

“Nick, I know that you know the literature in this space, and care about design and assessment. Can you say something about how you designed your MOOC to reach those who would not otherwise get access to formal educational opportunities? And since your MOOC has started, do you know yet if you achieved that goal — are you reaching people who would not otherwise get access?”

So here is that response. Thanks for the nudge, Mark! The answer is a bit long but please bear with me. We will be posting a longer summary after the course is completed, in a month or so. Consider this the unedited taster. I’m putting this here, early, prior to the detailed statistical work, so you can see where we are. All the numbers below are fresh off the system, to drive discussion and answering Mark’s question at, pretty much, a conceptual level.

First up, as some background for everyone, the MOOC team I’m working with is the University of Adelaide‘s Computer Science Education Research group, led by A/Prof Katrina Falkner, with me (Dr Nick Falkner), Dr Rebecca Vivian, and Dr Claudia Szabo.

I’ll start by noting that we’ve been working to solve the inherent scaling issues in the front of the classroom for some time. If I had a class of 12 then there’s no problem in engaging with everyone but I keep finding myself in rooms of 100+, which forces some people to sit away from me and also limits the number of meaningful interactions I can make to individuals in one setting. While I take Mark’s point about the front of the classroom, and the associated research is pretty solid on this, we encountered an inherent problem when we identified that students were better off down the front… and yet we kept teaching to rooms with more student than front. I’ll go out on a limb and say that this is actually a moral issue that we, as a sector, have had to look at and ignore in the face of constrained resources. The nature of large spaces and people, coupled with our inability to hover, means that we can either choose to have a row of students effectively in a semi-circle facing us, or we accept that after a relatively small number of students or number of rows, we have constructed a space that is inherently divided by privilege and will lead to disengagement.

So, Katrina’s and my first foray into this space was dealing with the problem in the physical lecture spaces that we had, with the 100+ classes that we had.

Katrina and I published a paper on “contributing student pedagogy” in Computer Science Education 22 (4), 2012, to identify ways for forming valued small collaboration groups as a way to promote engagement and drive skill development. Ultimately, by reducing the class to a smaller number of clusters and making those clusters pedagogically useful, I can then bring the ‘front of the class’-like experience to every group I speak to. We have given talks and applied sessions on this, including a special session at SIGCSE, because we think it’s a useful technique that reduces the amount of ‘front privilege’ while extending the amount of ‘front benefit’. (Read the paper for actual detail – I am skimping on summary here.)

We then got involved in the support of the national Digital Technologies curriculum for primary and middle school teachers across Australia, after being invited to produce a support MOOC (really a SPOC, small, private, on-line course) by Google. The target learners were teachers who were about to teach or who were teaching into, initially, Foundation to Year 6 and thus had degrees but potentially no experience in this area. (I’ve written about this before and you can find more detail on this here, where I also thanked my previous teachers!)

The motivation of this group of learners was different from a traditional MOOC because (a) everyone had both a degree and probable employment in the sector which reduced opportunistic registration to a large extent and (b) Australian teachers are required to have a certain number of professional development (PD) hours a year. Through a number of discussions across the key groups, we had our course recognised as PD and this meant that doing our course was considered to be valuable although almost all of the teachers we spoke to were furiously keen for this information anyway and my belief is that the PD was very much ‘icing’ rather than ‘cake’. (Thank you again to all of the teachers who have spent time taking our course – we really hope it’s been useful.)

To discuss access and reach, we can measure teachers who’ve taken the course (somewhere in the low thousands) and then estimate the number of students potentially assisted and that’s when it gets a little crazy, because that’s somewhere around 30-40,000.

In his talk at CSEDU 2014, Hugh Davis identified the student groups who get involved in MOOCs as follows. The majority of people undertaking MOOCs were life-long learners (older, degreed, M/F 50/50), people seeking skills via PD, and those with poor access to Higher Ed. There is also a small group who are Uni ‘tasters’ but very, very small. (I think we can agree that tasting a MOOC is not tasting a campus-based Uni experience. Less ivy, for starters.) The three approaches to the course once inside were auditing, completing and sampling, and it’s this final one that I want to emphasise because this brings us to one of the differences of MOOCs. We are not in control of when people decide that they are satisfied with the free education that they are accessing, unlike our strong gatekeeping on traditional courses.

I am in total agreement that a MOOC is not the same as a classroom but, also, that it is not the same as a traditional course, where we define how the student will achieve their goals and how they will know when they have completed. MOOCs function far more like many people’s experience of web browsing: they hunt for what they want and stop when they have it, thus the sampling engagement pattern above.

(As an aside, does this mean that a course that is perceived as ‘all back of class’ will rapidly be abandoned because it is distasteful? This makes the student-consumer a much more powerful player in their own educational market and is potentially worth remembering.)

Knowing these different approaches, we designed the individual subjects and overall program so that it was very much up to the participant how much they chose to take and individual modules were designed to be relatively self-contained, while fitting into a well-designed overall flow that built in terms of complexity and towards more abstract concepts. Thus, we supported auditing, completing and sampling, whereas our usual face-to-face (f2f) courses only support the first two in a way that we can measure.

As Hugh notes, and we agree through growing experience, marking/progress measures at scale are very difficult, especially when automated marking is not enough or not feasible. Based on our earlier work in contributing collaboration in the class room, for the F-6 Teacher MOOC we used a strong peer-assessment model where contributions and discussions were heavily linked. Because of the nature of the cohort, geographical and year-level groups formed who then conducted additional sessions and produced shared material at a slightly terrifying rate. We took the approach that we were not telling teachers how to teach but we were helping them to develop and share materials that would assist in their teaching. This reduced potential divisions and allows us to establish a mutually respectful relationship that facilitated openness.

(It’s worth noting that the courseware is creative commons, open and free. There are people reassembling the course for their specific take on the school system as we speak. We have a national curriculum but a state-focused approach to education, with public and many independent systems. Nobody makes any money out of providing this course to teachers and the material will always be free. Thank you again to Google for their ongoing support and funding!)

Overall, in this first F-6 MOOC, we had higher than usual retention of students and higher than usual participation, for the reasons I’ve outlined above. But this material was for curriculum support for teachers of young students, all of whom were pre-programming, and it could be contained in videos and on-line sharing of materials and discussion. We were also in the MOOC sweet-spot: existing degreed learners, PD driver, and their PD requirement depended on progressive demonstration on goal achievement, which we recognised post-course with a pre-approved certificate form. (Important note: if you are doing this, clear up how the PD requirements are met and how they need to be reported back, as early on as you can. It meant that we could give people something valuable in a short time.)

The programming MOOC, Think. Create. Code on EdX, was more challenging in many regards. We knew we were in a more difficult space and would be more in what I shall refer to as ‘the land of the average MOOC consumer’. No strong focus, no PD driver, no geographically guaranteed communities. We had to think carefully about what we considered to be useful interaction with the course material. What counted as success?

To start with, we took an image-based approach (I don’t think I need to provide supporting arguments for media-driven computing!) where students would produce images and, over time, refine their coding skills to produce and understand how to produce more complex images, building towards animation. People who have not had good access to education may not understand why we would use programming in more complex systems but our goal was to make images and that is a fairly universally understood idea, with a short production timeline and very clear indication of achievement: “Does it look like a face yet?”

In terms of useful interaction, if someone wrote a single program that drew a face, for the first time – then that’s valuable. If someone looked at someone else’s code and spotted a bug (however we wish to frame this), then that’s valuable. I think that someone writing a single line of correct code, where they understand everything that they write, is something that we can all consider to be valuable. Will it get you a degree? No. Will it be useful to you in later life? Well… maybe? (I would say ‘yes’ but that is a fervent hope rather than a fact.)

So our design brief was that it should be very easy to get into programming immediately, with an active and engaged approach, and that we have the same “mostly self-contained week” approach, with lots of good peer interaction and mutual evaluation to identify areas that needed work to allow us to build our knowledge together. (You know I may as well have ‘social constructivist’ tattooed on my head so this is strongly in keeping with my principles.) We wrote all of the materials from scratch, based on a 6-week program that we debated for some time. Materials consisted of short videos, additional material as short notes, participatory activities, quizzes and (we planned for) peer assessment (more on that later). You didn’t have to have been exposed to “the lecture” or even the advanced classroom to take the course. Any exposure to short videos or a web browser would be enough familiarity to go on with.

Our goal was to encourage as much engagement as possible, taking into account the fact that any number of students over 1,000 would be very hard to support individually, even with the 5-6 staff we had to help out. But we wanted students to be able to develop quickly, share quickly and, ultimately, comment back on each other’s work quickly. From a cognitive load perspective, it was crucial to keep the number of things that weren’t relevant to the task to a minimum, as we couldn’t assume any prior familiarity. This meant no installers, no linking, no loaders, no shenanigans. Write program, press play, get picture, share to gallery, winning.

As part of this, our support team (thanks, Jill!) developed a browser-based environment for Processing.js that integrated with a course gallery. Students could save their own work easily and share it trivially. Our early indications show that a lot of students jumped in and tried to do something straight away. (Processing is really good for getting something up, fast, as we know.) We spent a lot of time testing browsers, testing software, and writing code. All of the recorded materials used that development environment (this was important as Processing.js and Processing have some differences) and all of our videos show the environment in action. Again, as little extra cognitive load as possible – no implicit requirement for abstraction or skills transfer. (The AdelaideX team worked so hard to get us over the line – I think we may have eaten some of their brains to save those of our students. Thank you again to the University for selecting us and to Katy and the amazing team.)

The actual student group, about 20,000 people over 176 countries, did not have the “built-in” motivation of the previous group although they would all have their own levels of motivation. We used ‘meet and greet’ activities to drive some group formation (which worked to a degree) and we also had a very high level of staff monitoring of key question areas (which was noted by participants as being very high for EdX courses they’d taken), everyone putting in 30-60 minutes a day on rotation. But, as noted before, the biggest trick to getting everyone engaged at the large scale is to get everyone into groups where they have someone to talk to. This was supposed to be provided by a peer evaluation system that was initially part of the assessment package.

Sadly, the peer assessment system didn’t work as we wanted it to and we were worried that it would form a disincentive, rather than a supporting community, so we switched to a forum-based discussion of the works on the EdX discussion forum. At this point, a lack of integration between our own UoA programming system and gallery and the EdX discussion system allowed too much distance – the close binding we had in the R-6 MOOC wasn’t there. We’re still working on this because everything we know and all evidence we’ve collected before tells us that this is a vital part of the puzzle.

In terms of visible output, the amount of novel and amazing art work that has been generated has blown us all away. The degree of difference is huge: armed with approximately 5 statements, the number of different pieces you can produce is surprisingly large. Add in control statements and reputation? BOOM. Every student can write something that speaks to her or him and show it to other people, encouraging creativity and facilitating engagement.

From the stats side, I don’t have access to the raw stats, so it’s hard for me to give you a statistically sound answer as to who we have or have not reached. This is one of the things with working with a pre-existing platform and, yes, it bugs me a little because I can’t plot this against that unless someone has built it into the platform. But I think I can tell you some things.

I can tell you that roughly 2,000 students attempted quiz problems in the first week of the course and that over 4,000 watched a video in the first week – no real surprises, registrations are an indicator of interest, not a commitment. During that time, 7,000 students were active in the course in some way – including just writing code, discussing it and having fun in the gallery environment. (As it happens, we appear to be plateauing at about 3,000 active students but time will tell. We have a lot of post-course analysis to do.)

It’s a mistake to focus on the “drop” rates because the MOOC model is different. We have no idea if the people who left got what they wanted or not, or why they didn’t do anything. We may never know but we’ll dig into that later.

I can also tell you that only 57% of the students currently enrolled have declared themselves explicitly to be male and that is the most likely indicator that we are reaching students who might not usually be in a programming course, because that 43% of others, of whom 33% have self-identified as women, is far higher than we ever see in classes locally. If you want evidence of reach then it begins here, as part of the provision of an environment that is, apparently, more welcoming to ‘non-men’.

We have had a number of student comments that reflect positive reach and, while these are not statistically significant, I think that this also gives you support for the idea of additional reach. Students have been asking how they can save their code beyond the course and this is a good indicator: ownership and a desire to preserve something valuable.

For student comments, however, this is my favourite.

I’m no artist. I’m no computer programmer. But with this class, I see I can be both. #processingjs (Link to student’s work) #code101x .

That’s someone for whom this course had them in the right place in the classroom. After all of this is done, we’ll go looking to see how many more we can find.

I know this is long but I hope it answered your questions. We’re looking forward to doing a detailed write-up of everything after the course closes and we can look at everything.

CSEDU Day 1, Opening Panel, “Shaping the Future Learning Environment – Smart, Digital and Open?”

Only 32 papers out of 250 were accepted for the conference as full papers (12.8% acceptance, highly respectable) and were identified as being of “outstanding quality” – good work, PhD Student T and the CSER team! After the opening address, we went to the first panel, chaired by James Uhomobihi and Markus Helfert. The four keynote speakers are also on the panel but I’ll add more on them during their sessions.However, in summary, we had an academic, a maths instruction evangelist, a psychologist and a representative of engineering society bureaucracy (not as bad as it sounds). Everyone was saying how happy they were to be in Barcelona! (And who can blame them? This is one of my favourite cities and should be on everyone’s bucket list.)

Larissa Fradkin: “Mathematics Teaching: Is the future syncretic?”.

From Wikipedia: Syncretism /ˈsɪŋkrətɪzəm/ is the combining of different, often seemingly contradictory beliefs, while melding practices of various schools of thought.

Everyone was asked to give a position statement, which is a different take on panels for me, very interesting. Larissa had slides and she identified the old problem of the difficulty of teaching mathematics, alluding to the US mathematics wars. Are MOOCs the solution? We were promised a great deal yet much of it has not yet appeared. Well, Coursera materials appear to be useful, or useful in principle, but they don’t work in some classroom so they have to be localised. What is the role of the faculty member in this space? It’s a difficult question and the answer depends on time, teacher interest and student sophistication. We inherit the students that our preceding teachers have produced so text books and curricula have a big impact on the students that get turned out, if educational resources are presented in an unquestioned way. Trudging through exercises and content is one way to get through but what does it do? Does it teach? Does it prepare students for tests? Texts and resources are often, despite what publishers and authors claim, unaligned with the curriculum.

Schmidt’s last study (ref?) shows that quality teachers and quality materials are the top two considerations for inhaling student learning. We can produce well-crafted eBooks and MOOCs with editable and updatable content to give a flexible product – but the “rush to market” product doesn’t meet this requirement. The cognitive tutor was mentioned as AI-enriched educational software, under John Anderson’s model, but they are incredibly difficult to develop. There are tools out there that combine cognitive psychology and AI, CIRCLE and AutoTutor. However, most tool development is driven by psychologists and cognitive scientists rather than discipline experts and this can be a problem.

The speaker finished with a discussion on a semi-traditionalist semi-constructivist approach to rigorous instruction that required much less memorisation, focusing on conceptual understanding and developing more master than a straight constructivist approach. (Not quite sure of the details here but this will be extended in a later talk.)

Erik de Graffe gave his first slide from tomorrow on Team Learning in Engineering Education, starting with the statement “people are not born to work in a team”, which is an interesting statement given the entirety of human society. Students do like working together but the first meeting of the group can be challenging and they don’t know how to start. (Is this a cultural artefact based on the isolation and protection inherent in privilege? Time to get our Hofstader’s cultural dimensions hats on because this sounds a lot like a communal/separate categorical separation.) Erik noted that “team” is a word for animals harnessed together to apply more force in one direction – which is not what we want in a human team. (Although this is perhaps another cultural insight). Erik’s second statement is that communication is highly inaccurate: cog scientists estimate that the amount of information we process from the potential information around us is less than 1%. Most of the information that reaches our sense are ignored and yet we still make decisions. (Given I’m listening to the speaker while summarising another discussion and some slides, I’m wondering if this is the best way to express this while not identifying the difference in task focus and activities that are relevant to the task at hand.) Erik believes that on-line will make this worse. (I really need to go to his talk to see the evidence and caveats in all of this.) Erik then projected his own behaviour in on-line meetings and low attention to the general case – if Erik is in a meeting with you, get him to leave his camera on or he will be off making coffee. 🙂 Erik then moved on to virtual identities that allow people to do things in the alternative reality that they wouldn’t do in real life. Urm. I need to see the talk but this all seems a little dated to me but, hey, what do I know until I’ve seen the talk?

The next speaker, Steve, asked us if we shut the door while teaching (I don’t usually unless the noise get stop bad because it improves air flow, rather than for pedagogical or privacy reasons) but segued into a discussion on openness. Steve then referred to the strange issue of us providing free content to journal publishers that we then pay for. (I wrote a little something on this years ago for SIGCSE.) Steve then referred to the different time in closed and open access journals – the open access journal was “in print” in four weeks, versus the 18 months for the closed access, and the citation counts were more than an order of magnitude different. You can also measure the readers in an open access format. Open access materials are also crucial to scholars – Steve licenses all of his materials under Creative Commons so you can use his work and all you have to do is to acknowledge the source. When you open your content, you become an educator for the world. If people need education, then who are we to not provide the materials that they need? When you become an open scholar, you must prepare for criticism because many people will read things. You be also be ready for dialogue and discourse. It’s not always easy but it is very valuable. (I agree with this wholeheartedly.)

The next speaker, José Carlos Quadrado, President of IFEES, tries to infect people with good ideas about Engineering Education as part of his role. Is the future learning environment smart, digital and open? Talking about smart, an example is the new smart watch, a watch that is also a phone or linked to a device – what do we mean? When you have a smart phone, you have approximately 1.4 tons of technology from the 1980, which makes us wonder if smart means leaner? When we are going digital is this replacing paper with silicon? How do we handle factual authority when we have so much openly available where the traditional peer review and publishing oversight mechanisms are eroding and changing rapidly. (Another slightly creaky perspective although with a great deal of self-awareness.)

Teaching and learning tools have changed a great deal over the same time but have our pedagogical approaches also changed or been truly enhanced by this? There were some pretty broad generational (X vs Z) comparisons that I question the validity of. The notion that students of today couldn’t sit throughout this session is not something I agree with. Again, the message from the panel, with a couple of exceptions, is pretty dated and I have a bit of an urge to get off a lawn. Oh, and we finished with an Einstein quote after a name drop to a famous scientist. Look, I accept a lot of the things that are being said on the stage but we have to stop acting as if natural selection works in 18 months and that the increasing sophistication of later generations is anything more than an ability to make better choices because more and better choices are available. Oh, another Einstein quote.

I realise that I have started editorialising here, which I try not to do, but I am being bombarded with position statements that are leaden in their adherence to received wisdom on young people and those smart young. These issues are just clouding the real focus of the systems that we could use, the approaches we could take and the fact that for every highly-advanced Z Westerner in a low power-distance, highly self-centred approach, we have 100-1000 Pre-X non-Westerners in a high power-distance and communal environment.

We’re in question time. Erik asked about the ship analogy – when ships were made of wood, men were made of iron, and by moving to iron ships, we weakened people. Erik then went on to the question “Do smart phones make stupid people?” which basically nails down the coffin lid on all of the problems I’ve had with this opening. Steve, in response to another question, raised the connectivist argument for networking and distributed knowledge storage, which smart phones of course facilitate. Sadly, this foray into common sense was derailed by some sophistry on young people trying to be smart before they are clever.

There was a good point made that Universities will continue to be involved in quality education but they are no longer the bastions of information – that particular ship has sailed. Oh, there we go, we’ve dipped down again. Apparently we have now changed the way we buy things because we are now all concerned with perception. People are buying smart phones, not because they are smart, but because they want instant gratification. Generation Z are apparently going to be the generation that will reject everything and walk away. Eh, maybe.

Perhaps I shall come back to this later.