ITiCSE 2014, Day 3, Session 6A, “Digital Fluency”, #ITiCSE2014 #ITiCSE
Posted: June 25, 2014 Filed under: Education | Tags: ALICE, arm the princess, Bologna model, competency, competency-based assessment, computational thinking, computer science education, Duke, education, educational problem, educational research, empowering minorities, empowering women, higher education, ITiCSE, ITiCSE 2014, key competencies, learning, middle school, non-normative approaches, pattern analysis, principles of design, reflection, teaching, thinking, tools, women in computing Leave a comment
I’m at the Ångstrom Laboratory of Uppsala so this portrait hangs in the main hall. Hi, Anders!
The first paper was “A Methodological Approach to Key Competences in Informatics”, presented by Christina Dörge. The motivation for this study is moving educational standards from input-oriented approaches to output-oriented approaches – how students will use what you teach them in later life. Key competencies are important but what are they? What are the definitions, terms and real meaning of the words “key competencies”? A certificate of a certain grade or qualification doesn’t actually reflect true competency is many regards. (Bologna focuses on competencies but what do really mean?) Competencies also vary across different disciplines as skills are used differently in different areas – can we develop a non-normative approach to this?
The author discussed Qualitative Content Analysis (QCA) to look at different educational methods in the German educational system: hardware-oriented approaches, algorithm-oriented, application-oriented, user-oriented, information-oriented and, finally, system-oriented. The paradigm of teaching has shifted a lot over time (including the idea-oriented approach which is subsumed in system-oriented approaches). Looking across the development of the paradigms and trying to work out which categories developed requires a coding system over a review of textbooks in the field. If new competencies were added, then they were included in the category system and the coding started again. The resulting material could be referred to as “Possible candidates of Competencies in Informatics”, but those that are found in all of the previous approaches should be included as Competencies in Informatics. What about the key ones? Which of these are found in every part of informatics: theoretical, technical, practical and applied (under the German partitioning)? A key competency should be fundamental and ubiquitous.
The most important key competencies, by ranking, was algorithmic thinking, followed by design thinking, then analytic thinking (must look up the subtle difference here). (The paper contains all of the details) How can we gain competencies, especially these key ones, outside of a normative model that we have to apply to all contexts? We would like to be able to build on competencies, regardless of entry point, but taking into account prior learning so that we can build to a professional end point, regardless of starting point. What do we want to teach in the universities and to what degree?
The author finished on this point and it’s a good question: if we view our progression in terms of competency then how we can use these as building blocks to higher-level competencies? THis will help us in designing pre-requsitites and entry and exit points for all of our educational design.
The next talk was “Weaving Computing into all Middle School Disciplines”, presented by Susan Rodger from Duke. There were a lot of co-authors who were undergraduates (always good to see). The motivation for this project was there are problems with CS in the K-12 grades. It’s not taught in many schools and definitely missing in many high schools – not all Unis teach CS (?!?). Students don’t actually know what it is (the classic CS identify problem). There are also under-represented groups (women and minorities). Why should we teach it? 21st century skills, rewordings and many useful skills – from NCWIT.org.
Schools are already content-heavy so how do we convince people to add new courses? We can’t really so how about trying to weave it in to the existing project framework. Instead of doing a poster or a PowerPoint prevention, why not provide an animations that’s interactive in some way and that will involve computing. One way to achieve this is to use Alice, creating interactive stories or games, learning programming and computation concepts in a drag-and-drop code approach. Why Alice? There are many other good tools (Greenfoot, Lego, Scratch, etc) – well, it’s drag-and-drop, story-based and works well for women. The introductory Alice course in 2005 started to attract more women and now the class is more than 50% women. However, many people couldn’t come in because they didn’t have the prerequisites so the initiative moved out to 4th-6th grade to develop these skills earlier. Alice Virtual Worlds excited kids about computing, even at the younger ages.
The course “Adventures in Alice Programming” is aimed at grades 5-12 as Outreach, without having to use computing teachers (which would be a major restriction). There are 2-week teacher workshops where, initially, the teachers are taught Alice for a week, then the following week they develop lesson plans. There’s a one-week follow-up workshop the following summer. This initiative is funded until Summer, 2015, and has been run since 2008. There are sites: Durham, Charleston and Southern California. The teachers coming in are from a variety of disciplines.
How is this used on middle and high schools by teachers? Demonstrations, examples, interactive quizzes and make worlds for students to view. The students may be able to undertake projects, take and build quizzes, view and answer questions about a world, and the older the student, the more they can do.
Recruitment of teachers has been interesting. Starting from mailing lists and asking the teachers who come, the advertising has spread out across other conferences. It really helps to give them education credits and hours – but if we’re going to pay people to do this, how much do we need to pay? In the first workshop, paying $500 got a lot of teachers (some of whom were interested in Alice). The next workshop, they got gas money ($50/week) and this reduced the number down to the more interested teachers.
There are a lot of curriculum materials available for free (over 90 tutorials) with getting-started material as a one-hour tutorial showing basic set-up, placing objects, camera views and so on. There are also longer tutorials over several different stories. (Editor’s note: could we get away from the Princess/Dragon motif? The Princess says “Help!” and waits there to be rescued and then says “My Sweet Prince. I am saved.” Can we please arm the Princess or save the Knight?) There are also tutorial topics on inheritance, lists and parameter usage. The presenter demonstrated a lot of different things you can do with Alice, including book reports and tying Alice animations into the real world – such as boat trips which didn’t occur.
It was weird looking at the examples, and I’m not sure if it was just because of the gender of the authors, but the kitchen example in cooking with Spanish language instruction used female characters, the Princess/Dragon had a woman in a very passive role and the adventure game example had a male character standing in the boat. It was a small sample of the materials so I’m assuming that this was just a coincidence for the time being or it reflects the gender of the creator. Hmm. Another example and this time the Punnett Squares example has a grey-haired male scientist standing there. Oh dear.
Moving on, lots of helper objects are available for you to use if you’re a teacher to save on your development time which is really handy if you want to get things going quickly.
Finally, on discussing the impact, one 200 teachers have attend the workshops since 2008, who have then go on to teach 2900 students (over 2012-2013). From Google Analytics, over 20,000 users have accessed the materials. Also, a number of small outreach activities, Alice for an hour, have been run across a range of schools.
The final talk in this session was “Early validation of Computational Thinking Pattern Analysis”, presented by Hilarie Nickerson, from University of Colorado at Boulder. Computational thinking is important and, in the US, there have been both scope and pedagogy discussions, as well as instructional standards. We don’t have as much teacher education as we’d like. Assuming that we want the students to understand it, how can we help the teachers? Scalable Game Design integrates game and simulation design into public school curricula. The intention is to broaden participation for all kinds of schools as after-scjool classes had identified a lot of differences in the groups.
What’s the expectation of computational thinking? Administrators and industry want us to be able to take game knowledge and potentially use it for scientific simulation. A good game of a piece of ocean is also a predator-prey model, after all. Does it work? Well, it’s spread across a wide range of areas and communities, with more than 10,000 students (and a lot of different frogger games). Do they like it? There’s a perception that programming is cognitively hard and boring (on the congnitive/affective graph ranging from easy-hard/exciting-boring) We want it to be easy and exciting. We can make it easier with syntactic support and semantic support but making it exciting requires the students to feel ownership and to be able to express their creativity. And now they’re looking at the zone of proximal flow, which I’ve written about here. It’s good see this working in a project first, principles first model for these authors. (Here’s that picture again)
Figure from A. Repenning, “Programming Goes to School”, CACM, 55, 5, May, 2012.
The results? The study spanned 10,000 students, 45% girls and 55% boys (pretty good numbers!), 48% underrepresented, with some middle schools exposing 350 students per year. The motivation starts by making things achievable but challenging – starting from 2D basics and moving up to more sophisticated 3D games. For those who wish to continue: 74% boys, 64% girls and 69% of minority students want to continue. There are other aspects that can raise motivation.
What about the issue of Computing Computational Thinking? The authors have created a Computational Thinking Pattern Analysis (CTPA) instrument that can track student learning trajectories and outcomes. Guided discovery, as a pedagogy, is very effective in raising motivation for both genders, where direct instruction is far less effective for girls (and is also less effective for boys).
How do we validate this? There are several computational thinking patterns grouped using latent semantic analysis. One of the simpler patterns for a game is the pair generation and absorption where we add things to the game world (trucks in Frogger or fish in predator/prey) and then remove them (truck gets off the screen/fish gets eaten). We also need collision detection. Measuring skill development across these skills will allow you to measure it in comparison to the tutorial and to other students. What does CTPA actually measure? The presence of code patterns that corresponded to computational thinking constructs suggest student skill with computational thinking (but doesn’t prove it) and is different from measuring learning. The graphs produced from this can be represented as a single number, which is used for validation. (See paper for the calculation!)
This has been running for two years now, with 39 student grades for 136 games, with the two human graders shown to have good inter-rater consistency. Frogger was not very heavily correlated (Spearman rank) but Sokoban, Centipede and the Sims weren’t bad, and removing design aspects of rubrics may improve this.
Was their predictive validity in the project? Did the CTPA correlate with the skill score of the final game produced? Yes, it appears to be significant although this is early work. CTPA does appear to be cabal of measuring CT patterns in code that correlate with human skill development. Future work on this includes the refinement of CTPA by dealing with the issue of non-orthogonal constructs (collisions that include generative and absorptive aspects), using more information about the rules and examining alternative calculations. The group are also working not oils for teachers, including REACT (real-time visualisations for progress assessment) and recommend possible skill trajectories based on their skill progression.
Education and Paying Back (#AdelEd #CSER #DigitalTechnologies #acara #SAEdu)
Posted: March 22, 2014 Filed under: Education, Opinion | Tags: ACARA, advocacy, collaboration, community, cser, cser digital technologies, curriculum, design, digital education, digital technologies, education, educational problem, educational research, Generation Why, Google, higher education, learning, MOOC, Primary school, primary school teacher, principles of design, reflection, resources, school teachers, secondary school, sharing, teaching approaches, thinking, tools 2 CommentsOn Monday, the Computer Science Education Research Group and Google (oh, like you need a link) will release their open on-line course to support F-6 Primary school teachers in teaching the new Digital Technologies curriculum. We are still taking registrations so please go the course website if you want to sign up – or just have a look! (I’ve blogged about this recently as part of Science meets Parliament but you can catch it again here.) The course is open, on-line and free, released under Creative Commons so that the only thing people can’t do is to try and charge for it. We’re very excited and it’s so close to happening, I can taste it!
Here’s that link again – please, sign up!
I’m posting today for a few reasons. If you are a primary school teacher who wants help teaching digital technologies, we’d love to see you sign up and join our community of hundreds of other people who are thinking the same thing. If you know a primary school teacher, or are a principal for a primary school, and think that this would interest people – please pass it on! We are most definitely not trying to teach teachers how to teach (apart from anything else, what presumption!) but we’re hoping that what we provide will make it easier for teachers to feel comfortable, confident and happy with the new DT curriculum requirements which will lead to better experiences all ’round.
My other reason is one that came to me as I was recording my introduction section for the on-line course. In that brief “Oh, what a surprise there’s a camera” segment, I note that I consider the role of my teachers to have been essential in getting me to where I am today. This is what I’d like to do today: explicitly name and thank a few of my teachers and hope that some of what we release on Monday goes towards paying back into the general educational community.
You know who this is for.
My first thanks go to Mrs Shand from my Infant School in England. I was an early reader and, in an open plan classroom, she managed to keep me up with the other material while dealing with the fact that I was a voracious reader who would disappear to read at the drop of a hat. She helped to amplify my passion for reading, instead of trying to control it. Thank you!
In Australia, I ran into three people who were crucial to my development. Adam West was interested in everything so Grade 5 was full of computers (my first computing experience) because he arranged to borrow one and put it into the classroom in 1978, German (I can still speak the German I learnt in that class) and he also allowed us to write with nib and ink pens if we wanted – which was the sneakiest way to get someone’s handwriting and tidiness to improve that I have ever seen. Thank you, Adam! Mrs Lothian, the school librarian, also supported my reading habit and, after a while, all of the interesting books in the library often came through me very early on because I always returned them quickly and in good condition but this is where I was exposed to a whole world of interesting works: Nicholas Fisk, Ursula Le Guin and Susan Cooper not being the least of these. Thank you! Gloria Patullo (I hope I’ve spelt that correctly) was my Grade 7 teacher and she quickly worked out that I was a sneaky bugger on occasion and, without ever getting angry or raising a hand, managed to get me to realise that being clever didn’t mean that you could get away with everything and that being considerate and honest were the most important elements to alloy with smart. Thank you! (I was a pain for many years, dear reader, so this was a long process with much intervention.)
Moving to secondary school, I had a series of good teachers, all of whom tried to take the raw stuff of me and turn it into something that was happier, more useful and able to take that undirected energy in a more positive direction. I have to mention Ken Watson, Glenn Mulvihill, Mrs Batten, Dr Murray Thompson, Peter Thomas, Dr Riceman, Dr Bob Holloway, Milton Haseloff (I still have fossa, -ae, [f], ditch, burned into my brain) and, of course, Geoffrey Bean, headmaster, strong advocate of the thinking approaches of Edward de Bono and firm believer in the importance of the strength one needs to defend those who are less strong. Thank you all for what you have done, because it’s far too much to list here without killing the reader: the support, the encouragement, the guidance, the freedom to try things while still keeping a close eye, the exposure to thinking and, on occasion, the simple act of sitting me down to get me to think about what the heck I was doing and where I was going. The fact that I now work with some of them, in their continuing work in secondary education, is a wonderful thing and a reminder that I cannot have been that terrible. (Let’s just assume that, shall we? Moving on – rapidly…)
Of course, it’s not just the primary and secondary school teachers who helped me but they are the ones I want to concentrate on today, because I believe that the freedom and opportunities we offer at University are wonderful but I realise that they are not yet available to everyone and it is only by valuing, supporting and developing primary and secondary school education and the teachers who work so hard to provide it that we can go further in the University sector. We are lucky enough to be a juncture where dedicated work towards the national curriculum (and ACARA must be mentioned for all the hard work that they have done) has married up with an Industry partner who wants us all to “get” computing (Thank you, Google, and thank you so much, Sally and Alan) at a time when our research group was able to be involved. I’m a small part of a very big group of people who care about what happens in our schools and, if you have children of that age, you’ve picked a great time to send them to school. 🙂
I am delighted to have even a small opportunity to offer something back into a community which has given me so much. I hope that what we have done is useful and I can’t wait for it to start.
Enemies, Friends and Frenemies: Distance, Categorisation and Fun.
Posted: January 29, 2014 Filed under: Education | Tags: curriculum, data visualisation, design, education, educational problem, games, higher education, in the student's head, learning, principles of design, resources, student perspective, teaching, teaching approaches, thinking, tools Leave a commentAs Mario Puzo and Francis Ford Coppola wrote in “The Godfather Part II”:
… keep your friends close but your enemies closer.
(I bet you thought that was Sun Tzu, the author of “The Art of War”. So did I but this movie is the first use.)
I was thinking about this the other day and it occurred to me that this is actually a simple modelling problem. Can I build a model which will show the space around me and where I would expect to find friends and enemies? Of course, you might be wondering “why would you do this?” Well, mostly because it’s a little bit silly and it’s a way of thinking that has some fun attached to it. When I ask students to build models of the real world, where they think about how they would represent all of the important aspects of the problem and how they would simulate the important behaviours and actions seen with it, I often give them mathematical or engineering applications. So why not something a little more whimsical?
From looking at the quote, we would assume that there is some distance around us (let’s call it a circle) where we find everyone when they come up to talk to us, friend or foe, and let’s also assume that the elements “close” and “closer” refer to how close we let them get in conversation. (Other interpretations would have us living in a neighbourhood of people who hate us, while we have to drive to a different street to sit down for dinner with people who like us.) So all of our friends and enemies are in this circle, but enemies will be closer. That looks like this:
I have more friends than enemies because I’m popular!
So now we have a visual model of what is going on and, if we wanted to, we could build a simple program that says something like “if you’re in this zone, then you’re an enemy, but if you’re in that zone then you’re a friend” where we define the zones in terms of nested circular regions. But, as we know, friend always has your back and enemies stab you in the back, so now we need to add something to that “ME” in the middle – a notion of which way I’m facing – and make sure that I can always see my enemies. Let’s make the direction I’m looking an arrow. (If I could draw better, I’d put glasses on the front. If you’re doing this in the classroom, an actual 3D dummy head shows position really well.) That looks like this:
Same numbers but now I can keep an eye on those enemies!
Now our program has to keep track of which way we’re facing and then it checks the zones, on the understanding that either we’re going to arrange things to turn around if an enemy is behind us, or we can somehow get our enemies to move (possibly by asking nicely). This kind of exercise can easily be carried out by students and it raises all sorts of questions. Do I need all of my enemies to be closer than my friends or is it ok if the closest person to me is an enemy? What happens if my enemies are spread out in a triangle around me? Is they won’t move, do I need to keep rotating to keep an eye on them or is it ok if I stand so that they get as much of my back as they can? What is an acceptable solution to this problem? You might be surprised how much variation students will suggest in possible solutions, as they tell you what makes perfect sense to them for this problem.
When we do this kind of thing with real problems, we are trying to specify the problem to a degree that we remove all of the unasked questions that would otherwise make the problem ambiguous. Of course, even the best specification can stumble if you introduce new information. Some of you will have heard of the term ‘frenemy’, which apparently:
can refer to either an enemy pretending to be a friend or someone who really is a friend but is also a rival (from Wikipedia and around since 1953, amazingly!)
What happens if frenemies come into the mix? Well, in either case, we probably want to treat them like an enemy. If they’re an enemy pretending to be a friend, and we know this, then we don’t turn our back on them and, even in academia, it’s never all that wise to turn your back on a rival, either. (Duelling citations at dawn can be messy.) In terms of our simple model, we can deal with extending the model because we clearly understand what the important aspects are of this very simple situation. It would get trickier if frenemies weren’t clearly enemies and we would have to add more rules to our model to deal with this new group.
This can be played out with students of a variety of ages, across a variety of curricula, with materials as simple as a board, a marker and some checkers. Yet this is a powerful way to explain models, specification and improvement, without having to write a single line of actual computer code or talk about mathematics or bridges! I hope you found it useful.
“Begrudgingly honest because we might be surveilled?”
Posted: January 3, 2014 Filed under: Education, Opinion | Tags: blogging, community, design, education, ethics, Generation Why, higher education, O'Reilly Community, online conference, panopticon, principles of design, resources, thinking, tools, universal principles of design 1 Comment
The Plans of the Panopticon
O’Reilly Community are hosting an online conference on “Data, Crime, and Conflict”, which I’m attending at the rather unhealthy hour of 3:30am on the morning of January the 8th (it’s better for you if you’re in the UK or US). Here’s an extract of the text:
A world of sensors gives us almost complete surveillance. Every mobile device tracks moves, forming a digital alibi or new evidence for the prosecution. And with the right data, predictions look frighteningly like guilt.
How does a data-driven, connected world deal with crime, conflict, and peacekeeping? Will we be prisoners in a global Panopticon, begrudgingly honest because we might be surveilled? Or will total transparency even the balance between the enforcer and the citizen?
Join a lineup of thinkers and technologists for this free online event as we look at the ways data is shaping how we police ourselves, from technological innovations to ethical dilemmas.
I’ve been interested in the possible role and expansion (and the implications) of the panopticon since first reading about it. I even wrote a short story once to explore a global society where the removal of privacy had not been the trip down into dystopia that we always expect it to be. (This doesn’t mean that I believe that it is a panacea – I just like writing stories!) I’m looking forward to seeing what the speakers have to say. They claim that there are limited places but I managed to sign up today so it’s probably not too late.
Skill Games versus Money Games: Disguising One Game As Another
Posted: July 5, 2013 Filed under: Education | Tags: advocacy, blogging, community, curriculum, design, education, educational problem, educational research, ethics, feedback, games, Generation Why, higher education, in the student's head, learning, plagiarism, principles of design, reflection, resources, teaching, teaching approaches, thinking, tools, universal principles of design Leave a commentI recently ran across a very interesting article on Gamasutra on the top tips for turning a Free To Play (F2P) game into a Paying game by taking advantage of the way that humans think and act. F2P games are quite common but, obviously, it costs money to make a game so there has to be some sort of associated revenue stream. In some cases, the F2P is a Lite version of the pay version, so after being hooked you go and buy the real thing. Sometimes there is an associated advertising stream, where you viewing the ads earns the producer enough money to cover costs. However, these simple approaches pale into insignificance when compared with the top tips in the link.
Ramin identifies two games for this discussion: games of skill, where it is your ability to make sound decisions that determines the outcome, and money games, where your success is determined by the amount of money you can spend. Games of chance aren’t covered here but, given that we’re talking about motivation and agency, we’re depending upon one specific blindspot (the inability of humans to deal sensibly with probability) rather than the range of issues identified in the article.
I dont want to rehash the entire article but the key points that I want to discuss are the notion of manipulating difficulty and fun pain. A game of skill is effectively fun until it becomes too hard. If you want people to keep playing then you have to juggle the difficulty enough to make it challenging but not so hard that you stop playing. Even where you pay for a game up front, a single payment to play, you still want to get enough value out of it – too easy and you finish too quickly and feel that you’ve wasted your money; too hard and you give up in disgust, again convinced that you’ve wasted your money. Ultimately, in a pure game of skill, difficulty manipulation must be carefully considered. As the difficulty ramps up, the player is made uncomfortable, the delightful term fun pain is applied here, and resolving the difficulty removes this.
Or, you can just pay to make the problem go away. Suddenly your game of skill has two possible modes of resolution: play through increasing difficulty, at some level of discomfort or personal inconvenience, or, when things get hard enough, pump in a deceptively small amount of money to remove the obstacle. The secret of the P2P game that becomes successfully monetised is that it was always about the money in the first place and the initial rounds of the game were just enough to get you engaged to a point where you now have to pay in order to go further.
You can probably see where I’m going with this. While it would be trite to describe education as a game of skill, it is most definitely the most apt of the different games on offer. Progress in your studies should be a reflection of invested time in study, application and the time spent in developing ideas: not based on being ‘lucky’, so the random game isn’t a choice. The entire notion of public education is founded on the principle that educational opportunities are open to all. So why do some parts of this ‘game’ feel like we’ve snuck in some covert monetisation?
I’m not talking about fees, here, because that’s holding the place of the fee you pay to buy a game in the first place. You all pay the same fee and you then get the same opportunities – in theory, what comes out is based on what the student then puts in as the only variable.
But what about textbooks? Unless the fee we charge automatically, and unavoidably, includes the cost of the textbook, we have now broken the game into two pieces: the entry fee and an ‘upgrade’. What about photocopying costs? Field trips? A laptop computer? An iPad? Home internet? Bus fare?
It would be disingenuous to place all of this at the feet of public education – it’s not actually the fault of Universities that financial disparity exists in the world. It is, however, food for thought about those things that we could put into our courses that are useful to our students and provide a paid alternative to allow improvement and progress in our courses. If someone with the textbook is better off than someone without the textbook, because we don’t provide a valid free alternative, then we have provided two-tiered difficulty. This is not the fun pain of playing a game, we are now talking about genuine student stress, a two-speed system and a very high risk that stressed students will disengage and leave.
From my earlier discussions on plagiarism, we can easily tie in Ramin’s notion of the driver of reward removal, where players have made so much progress that, on facing defeat, they will pay a fee to reduce the impact of failure; or, in some cases, to remove it completely. As Ramin notes:
“This technique alone is effective enough to make consumers of any developmental level spend.”
It’s not just lost time people are trying to get back, it’s the things that have been achieved in that time. Combine that with, in our case, the future employability and perception of that piece of paper, and we have a very strong behavioural driver. A number of the tricks Ramin describes don’t work as well on mature and aware thinkers but this one is pretty reliable. If it’s enough to make people pay money, regardless of their development level, then there are lots of good design decisions we can make from this – lower risk assessment, more checkpointing, steady progress towards achievement. We know lots of good ways to avoid this, if we consider it to be a problem and want to take the time to design around it.
This is one of the greatest lessons I’ve learned about studying behaviour, even as a rank amateur. Observing what people do and trying to build systems that will work despite that makes a lot more sense than building a system that works to some ideal and trying to jam people into it. The linked article shows us how people are making really big piles of money by knowing how people work. It’s worth looking at to make sure that we aren’t, accidentally, manipulating students in the same way.
The defining question.
Posted: July 2, 2013 Filed under: Education, Opinion | Tags: advocacy, community, education, educational problem, ethics, feedback, higher education, in the student's head, reflection, student perspective, teaching, teaching approaches, thinking, tools, workload Leave a commentThere has been a lot going on for me recently. A lot of thinking, a lot of work and an amount of getting involved in things because my students trust me and will come to me to ask questions, which sometimes puts me in the uncomfortable position of having to juggle my accommodation for the different approaches of my colleagues and my own beliefs, as well as acting in everyone’s best interests. I’m not going to go into details but I think that I can summarise my position on everything, as an educator, by phrasing it in one question.
Is this course of action to the student’s benefit?
I mean, that’s it, isn’t it? If the job is educating students and developing the citizens of tomorrow, then everything that we do should be to the benefit of the student and/or future graduate. But it’s never simple, is it, because the utilitarian calculus to derive benefit quickly becomes complicated when we consider the effect of institutional reputation or perception on the future benefit to the student. But maybe that’s over thinking things (gasp, I hear regular readers cry). I’m not sure I know how to guide student behaviour to raise my University’s ranking in various measures – but I do know how to guide student behaviour to reduce the number of silly or thoughtless things they do, to enhance their learning and to help them engage. Maybe the simple question is the best? Will the actions I take today improve my students’ knowledge or enhance their capacity to learn? Have I avoided wasting their time doing something that we do because we have always done it, rather than giving them something to do because it is what we should be doing? Am I always considering the benefit to the largest group of students, while considering the needs of the individual?
Every time I see a system that has a fixed measure of success, people optimise for it. If it’s maximum profit, people maximise profit. If it’s minimum space, people cut their space. Guidelines help a lot in working out which course of action to take: when faced with a choice between A and B, choose the option that maximises your objective. This even works without a strong vision of the future, which is good because I’m not sure we have a clear enough view of the long path to graduation to really be specific about this. There is always a risk that people will get the assessment of benefit wrong, which can lead to soft marking or lax standards, but I’m not a believer that post hoc harshness is the solution to inherited laxity from another system (especially where that may be a perception that’s not grounded in reality). Looking at all of my actions in terms of a real benefit, to the student, to their community, to our equality standards, to our society – that shines a bright light on what we do so we can clearly see what we’re doing and, if it requires change, illuminates the path to change.
Let’s not turn “Chalk and Talk” into “Watch and Scratch”
Posted: July 1, 2013 Filed under: Education | Tags: advocacy, authenticity, blogging, community, curriculum, design, education, educational problem, educational research, ethics, feedback, higher education, moocs, principles of design, reflection, resources, teaching, teaching approaches, tools, universal principles of design 1 CommentWe are now starting to get some real data on what happens when people “take” a MOOC (via Mark’s blog). You’ll note the scare quotes around the word “take”, because I’m not sure that we have really managed to work out what it means to get involved in a course that is offered through the MOOC mechanism. Or, to be more precise, some people think they have but not everyone necessarily agrees with them. I’m going to list some of my major concerns, even in the face of the new clickstream data, and explain why we don’t have a clear view of the true value/approaches for MOOCs yet.
- On-line resources are not on-line courses and people aren’t clear on the importance of an overall educational design and facilitation mechanism. Many people have mused on this in the past. If all the average human needed was a set of resources and no framing or assistive pedagogy then our educational resources would be libraries and there would be no teachers. While there are a number of offerings that are actually courses, applying the results of the MIT 6.002x to what are, for the most part, unstructured on-line libraries of lecture recordings is not appropriate. (I’m not even going to get into the cMOOC/xMOOC distinction at this point.) I suspect that this is just part of the general undervaluing of good educational design that rears its head periodically.
- Replacing lectures with on-line lectures doesn’t magically improve things. The problem with “chalk and talk”, where it is purely one-way with no class interaction, is that we know that it is not an effective way to transfer knowledge. Reading the textbook at someone and forcing them to slowly transcribe it turns your classroom into an inefficient, flesh-based photocopier. Recording yourself standing in front a class doesn’t automatically change things. Yes, your students can time shift you, both to a more convenient time and at a more convenient speed, but what are you adding to the content? How are you involving the student? How can the student benefit from having you there? When we just record lectures and put them up there, then unless they are part of a greater learning design, the student is now sitting in an isolated space, away from other people, watching you talk, and potentially scratching their head while being unable to ask you or anyone else a question. Turning “chalk and talk” into “watch and scratch” is not an improvement. Yes, it scales so that millions of people can now scratch their heads in unison but scaling isn’t everything and, in particular, if we waste time on an activity under the illusion that it will improve things, we’ve gone backwards in terms of quality for effort.
- We have yet to establish the baselines for our measurement. This is really important. An on-line system us capable of being very heavily tracked and it’s not just links. The clickstream measurements in the original report record what people clicked on as they worked with the material. But we can only measure that which is set up for measurement – so it’s quite hard to compare the activity in this course to other activities that don’t use technology. But there are two subordinate problems to this (and I apologise to physicists for the looseness of the following) :
- Heisenberg’s MOOC: At the quantum scale, you can either tell where something is or what it is doing – the act of observation has limits of precision. Borrowing that for the macro scale: measure someone enough and you’ll see how they behave under measurement but the measurements we pick tend to fall into the stage they’ve reached or the actions they’ve taken. It’s very complex to combine quantitative and qualitative measures to be able to map someone’s stage and their comprehension/intentions/trajectory. You don’t have to accept arguments based on the Hawthorne Effect to understand why this does not necessarily tell you much about unobserved people. There are a large number of people taking these courses out of curiosity, some of whom already have appropriate qualifications, with only 27% the type of student that you would expect to see at this level of University. Combine that with a large number of researchers and curious academics who are inspecting each other’s courses, I know of at least 12 people in my own University taking MOOCs of various kinds to see what they’re like, and we have the problem that we are measuring people who are merely coming in to have a look around and are probably not as interested in the actual course. Until we can actually shift MOOC demography to match that of our real students, we are always going to have our measurements affected by these observers. The observers might not mind being heavily monitored and observed, but real students might. Either way, numbers are not the real answer here – they show us what but there is still too much uncertainty in the why and the how.
- Schrödinger’s MOOC: Oh, that poor reductio ad absurdum cat. Does the nature of the observer change the behaviour of the MOOC and force it to resolve one way or another (successful/unsuccessful)? If so, how and when? Does the fact of observation change the course even more than just in enrolments and uncertainty of validity of figures? The clickstream data tells us that the forums are overwhelmingly important to students, with 90% of people who viewed threads without commenting, and only 3% of total students enrolled every actually posted anything in a thread. What was the make-up of that 3% and was it actual students or the over-qualified observers who then provided an environment that 90% of their peers found useful?
- Numbers need context and unasked questions give us no data: As one example, the authors of the study were puzzled that so few people had logged in from China, which surprised them. Anyone who has anything to do with network measurement is going to be aware that China is almost always an outlier in network terms. My blog, for example, has readers from around the world – but not China. It’s also important to remember that any number of Chinese network users will VPN/SSH to hosts outside China to enjoy unrestricted search and network access. There may have been many Chinese people (who didn’t self-identify for obvious reasons) who were using proxies from outside China. The numbers on this particular part of the study do not make sense unless they are correctly contextualised. We also see a lack of context in the reporting on why people were doing the course – the numbers for why people were doing it had to be augmented from comments in the forum that people ‘wanted to see if they could make it through an MIT course’. Why wasn’t that available from the initial questions?
- We don’t know what pass/fail is going to look like in this environment. I can’t base any MOOC plans of my own on how people respond to a MIT-branded course but it is important to note that MIT’s approach was far more than “watch and scratch”, as is reflected by their educational design in providing various forms of materials, discussions forums, homework and labs. But still, 155,000 people signed up for this and only 7,000 received certificates. 2/3 of people who registered then went on to do nothing. I don’t think that we can treat a success rate of less than 5% as a success rate. Even where we say that 2/3 dropped out, this still equates to a pass rate under 14%. Is that good? Is that bad? Taking everything into account from above, my answer is “We don’t know.” If we get 17% next time, is that good or bad? How do we make this better?
- The drivers are often wrong. Several US universities have gone on the record to complain about undermining their colleagues and have refused to take part in MOOC-related activities. The reasons for this vary but the greatest fear is that MOOCs will be used to reduce costs by replacing existing lecturing staff with a far smaller group and using MOOCs to handle the delivery. From a financial argument, MOOCs are astounding – 155,000 people contacted for the cost of a few lecturers. Contrast that with me teaching a course to 100 students. If we look at it from a quality perspective, and dealing with all of the points so far, we have no argument to say that MOOCs are as good as our good teaching – but we do know that they are easily as good as our bad teaching. But from a financial perspective? MOOC is king. That is, however, not how we guarantee educational quality. Of course, when we scale, we can maintain quality by increasing resources but this runs counter to a cost-saving argument so we’re almost automatically being prevented from doing what is required to make the large scale course work by the same cost driver that led to its production in the first place!
- There are a lot of statements but perhaps not enough discussion. These are trying times for higher education and everyone wants an edge, more students, higher rankings, to keep their colleagues and friends in work and, overall, to do the right thing for their students. Senior management, large companies, people worried about money – they’re all talking about MOOCs as if they are an accepted substitute for traditional approaches – at the same time as we are in deep discussion about which of the actual traditional approaches are worthwhile and which new approaches are going to work better. It’s a confusing time as we try to handle large-scale adoption of blended learning techniques at the same time people are trying to push this to the large scale.
I’m worried that I seem to be spending most of my time explaining what MOOCs are to people who are asking me why I’m not using a MOOC. I’m even more worried when I am still yet to see any strong evidence that MOOCs are going to provide anything approaching the educational design and integrity that has been building for the past 30 years. I’m positively terrified when I see corporate providers taking over University delivery before we have established actual measurable quality and performance guidelines for this incredibly important activity. I’m also bothered by statements found at the end of the study, which was given prominence as a pull quote:
[The students] do not follow the norms and rules that have governed university courses for centuries nor do they need to.
I really worry about this because I haven’t yet seen any solid evidence that this is true, yet this is exactly the kind of catchy quote that is going to be used on any number of documents that will come across my desk asking me when I’m going to MOOCify my course, rather than discussing if and why and how we will make a transition to on-line blended learning on the massive scale. The measure of MOOC success is not the number of enrolees, nor is it the number of certificates awarded, nor is it the breadth of people who sign up. MOOCs will be successful once we have worked out how to use this incredibly high potential approach to teaching to deliver education at a suitably high level of quality to as many people as possible, at a reduced or even near-zero cost. The potential is enormous but, right now, so is the risk!
Another semester, more lessons learned (mostly by me).
Posted: June 16, 2013 Filed under: Education, Opinion | Tags: advocacy, authenticity, collaboration, community, curriculum, design, education, educational problem, educational research, ethics, feedback, Generation Why, higher education, in the student's head, learning, plagiarism, principles of design, reflection, resources, student perspective, teaching, teaching approaches, thinking, tools, universal principles of design Leave a commentI’ve just finished the lecturing component for my first year course on programming, algorithms and data structures. As always, the learning has been mutual. I’ve got some longer posts to write on this at some time in the future but the biggest change for this year was dropping the written examination component down and bringing in supervised practical examinations in programming and code reading. This has given us some interesting results that we look forward to going through, once all of the exams are done and the marks are locked down sometime in late July.
Whenever I put in practical examinations, we encounter the strange phenomenon of students who can mysteriously write code in very short periods of time in a practical situation very similar to the practical examination, but suddenly lose the ability to write good code when they are isolated from the Internet, e-Mail and other people’s code repositories. This is, thank goodness, not a large group (seriously, it’s shrinking the more I put prac exams in) but it does illustrate why we do it. If someone has a genuine problem with exam pressure, and it does occur, then of course we set things up so that they have more time and a different environment, as we support all of our students with special circumstances. But to be fair to everyone, and because this can be confronting, we pitch the problems at a level where early achievement is possible and they are also usually simpler versions of the types of programs that have already been set as assignment work. I’m not trying to trip people up, here, I’m trying to develop the understanding that it’s not the marks for their programming assignments that are important, it’s the development of the skills.
I need those people who have not done their own work to realise that it probably didn’t lead to a good level of understanding or the ability to apply the skill as you would in the workforce. However, I need to do so in a way that isn’t unfair, so there’s a lot of careful learning design that goes in, even to the selection of how much each component is worth. The reminder that you should be doing your own work is not high stakes – 5-10% of the final mark at most – and builds up to a larger practical examination component, worth 30%, that comes after a total of nine practical programming assignments and a previous prac exam. This year, I’m happy with the marks design because it takes fairly consistent failure to drop a student to the point where they are no longer eligible for redemption through additional work. The scope for achievement is across knowledge of course materials (on-line quizzes, in-class scratchy card quizzes and the written exam), programming with reference materials (programming assignments over 12 weeks), programming under more restricted conditions (the prac exams) and even group formation and open problem handling (with a team-based report on the use of queues in the real world). To pass, a student needs to do enough in all of these. To excel, they have to have a good broad grasp of theoretical and practical. This is what I’ve been heading towards for this first-year course, a course that I am confident turns out students who are programmers and have enough knowledge of core computer science. Yes, students can (and will) fail – but only if they really don’t do enough in more than one of the target areas and then don’t focus on that to improve their results. I will fail anyone who doesn’t meet the standard but I have no wish to do any more of that than I need to. If people can come up to standard in the time and resource constraints we have, then they should pass. The trick is holding the standard at the right level while you bring up the people – and that takes a lot of help from my colleagues, my mentors and from me constantly learning from my students and being open to changing the learning design until we get it right.
Of course, there is always room for improvement, which means that the course goes back up on blocks while I analyse it. Again. Is this the best way to teach this course? Well, of course, what we will do now is to look at results across the course. We’ll track Prac Exam performance across all practicals, across the two different types of quizzes, across the reports and across the final written exam. We’ll go back into detail on the written answers to the code reading question to see if there’s a match for articulation and comprehension. We’ll assess the quality of response to the exam, as well as the final marked outcome, to tie this back to developmental level, if possible. We’ll look at previous results, entry points, pre-University marks…
And then we’ll teach it again!
“Hi, my name is Nick and I specialise in failure.”
Posted: June 10, 2013 Filed under: Education, Opinion | Tags: advocacy, collaboration, community, curriculum, design, education, educational research, ethics, failure, Generation Why, higher education, in the student's head, learning, measurement, reflection, resources, student perspective, survivorship, teaching, teaching approaches, thinking, tools Leave a commentI recently read an article on survivorship bias in the “You Are Not So Smart” website, via Metafilter. While the whole story addressed the World War II Statistical Research Group, it focused on the insight contributed by Abraham Wald, a statistician. The World War II Allied bomber losses were large, very large, and any chances of reducing this loss was incredibly valuable. The question was “How could the US improve their chances of bringing their bombers back intact?” Bombers landing back after missions were full of holes but armour just can’t be strapped willy-nilly on to a plane without it becoming land-locked. (There’s a reason that birds are so light!) The answer, initially, was obvious – find the place where the most holes were, by surveying the fleet, and patching them. Put armour on the colander sections and, voila, increased survival rate.
No, said Wald. That wouldn’t help.
Wald’s logic is both simple and convincing. If a plane was coming back with those holes in place, then the holes in the skin were not leading to catastrophic failure – they couldn’t have been if the planes were returning! The survivors were not showing the damage that would have led to them becoming lost aircraft. Wald used the already collected information on the damage patterns to work out how much damage could be taken on each component and the likelihood of this occurring during a bombing run. based on what kind of forces it encountered.
It’s worth reading the entire article because it’s a simple and powerful idea – attributing magical properties to the set of steps taken by people who have become ultra-successful is not going to be as useful as looking at what happened to take people out of the pathway to success. If you’ve read Steve Jobs’ biography then you’re aware that he had a number of interesting traits, only some of which may have led to him becoming as successful as he did. Of course, if you’ve been reading a lot, you’ll be aware of the importance of Paul Jobs, Steve Wozniak, Robert Noyce, Bill Gates, Jony Ive, John Lasseter, and, of course, his wife, Laurene Powell Jobs. So the whole “only eating fruit” thing, the “reality distortion field” thing and “not showering” thing (some of which he changed, some he didn’t) – which of these are the important things? Jobs, like many successful people, failed at some of his endeavours, but never in a way that completely wiped him out. Obviously. Now, when he’s not succeeding, he’s interesting, because we can look at the steps that took him down and say “Oh, don’t do that”, assuming that it’s something that can be changed or avoided . When he’s succeeding, there are so many other things getting in the way that depend upon what’s happened to you so far, who your friends are, and how many resources you get to play with, it’s hard to be able to give good advice on what to do.
I have been studying failure for some time. Firstly in myself, and now in my students. I look for those decisions, or behaviours, that lead to students struggling in their academic achievement, or to falling away completely in some cases. The majority of the students who come to me with a high level of cultural, financial and social resources are far less likely to struggle because, even when faced with a set-back, they rarely hit the point where they can’t bounce back – although, sadly, it does happen but in far fewer numbers. When they do fall over, it is for the same reasons as my less-advantaged students, who just do so in far greater numbers because they have less resilience to the set-backs. By studying failure, and the lessons learned and the things to be avoided, I can help all of my students and this does not depend upon their starting level. If I were studying the top 5% of students, especially those who had never received a mark less than A+, I would be surprised if I could learn much that I could take and usefully apply to those in the C- bracket. The reverse, however? There’s gold to be mined there.
By studying the borderlines and by looking for patterns in the swirling dust left by those departing, I hope that I can find things which reduce failure everywhere – because every time someone fails, we run the risk of not getting them back simply because failure is disheartening. Better yet, I hope to get something that is immediately usable, defensible and successful. Probably rather a big ask for a protracted study of failure!
Nick Falkner 