ICER 2012 Research Paper Session 2

Posted: September 13, 2012 | Author: | Filed under: Education | Tags: , , , , , , , , , , , , | 3 Comments

Ok, true confession time. My (and Katrina’s) paper was in this session and I’ll write this up separately. So this session consisted of “Adapting Disciplinary Commons Model: Lessons and Results from Georgia” (Brianna Morrison, Lijun Ni and Mark Guzdial) and… another paper. 🙂

The goals of the original disciplinary commons were:
  • To document and share knowledge about student learning in CS classrooms
  • To establish practices for the scholarship of teaching by making it public, peer-reviewed and amenable for public use. (portfolio model)
While the first goal was achieved, the second wasn’t as, although portfolios were produced, people just wanted to keep them private. However, they did
develop a strong and vibrant community, with associated change of practice as a result of participation. The next stage was a Disciplinary Commons for Computing Educators (DCCE, for Georgia), with the adaptation that this apply to both High School teachers AND university-level educators.
The new goals were:
  1. Creating community
  2. Sharing resources and knowledge of how things are taught in other contexts.
  3. Supporting student recruitment within the high school environment.
I was interested to learn that there is no Computer Science teaching certificate in Georgia, hence a teacher must be certified in another discipline, such as Mathematics, Science, and Business being most likely. (I believe this is what was said although on reviewing my notes, I find this a little confusing. I’m assuming that this is due to the transition into the Georgia teaching framework.)
The community results were very interesting, as the initial community formed where one person was the hub – a network but not a robust one! After working on this in year 3, a much more evenly distributed group was formed that could survive a few people dropping out.  Given that many of the students in the program had no (or very few) peers in the home university, these networks were crucial to giving them important information. Teachers who work in isolation need supporting networks – you can see what else someone does, and ask how they do it.
I love these community-building projects and the network example gave one of the fantastic insights into why regular progress and impact checks can make the difference between an ok project and a highly successful one. Identifying that a network based on one (hub) person is unstable and altering your practices to make the network graph more heavily meshed is an excellent adaptation that reinforces the key focus on this project: creating community.
I read a design magazine called Desktop, much to the amusement of my more design-oriented friends, and one of the smaller regular features is that of the desks and working environments of professional designers. As I try to learn more about this area, this helps give me some insight into the community and, because of this, it accelerates my development. The community project described in this paper allows people who are already trying to hold a presence in a tricky and evolving area by connecting them with people who have similar issues and joining all of them to a shared repository of knowledge and experience. It would be great to see more programs like this.

ICER 2012 Research Paper Session 1

Posted: September 13, 2012 | Author: | Filed under: Education | Tags: , , , , , , , , , , , | Leave a comment

It would not be over-stating the situation to say that every paper presented at ICER led to some interesting discussion and, in some cases, some more… directed discussion than others. This session started off with a paper entitled “Threshold Concepts and Threshold Skills in Computing” (Kate Sanders, Jonas Boustedt, Anna Eckerdal, Robert McCartney, Jan Erik Moström Lynda Thomas and Carol Zander), on whether threshold skills, as distinct from threshold concepts, existed and, if they did, what their characteristics would be. Threshold skills were described as transformative, integrative, troublesome knowledge, semi-irreversible (in that they’re never really lost), and requiring practice to keep current. The discussion that followed raised a lot of questions, including whether you could learn a skill by talking about it or asking someone – skill transfer questions versus environment. The consensus, as I judged it from the discussion, was that threshold skills didn’t follow from threshold concepts but there was a very rapid and high-level discussion that I didn’t quite follow, so any of the participants should feel free to leap in here!

The next talk was “On the reliability of Classifying Programming Tasks Using a Neo-Piagetian Theory of Cognitive Development” (Richard Gluga, Raymond Lister, Judy Kay, Sabina Kleitman and Donna Teague), where Ray raised and extended a number of the points that he had originally shared with us in the workshop on Sunday. Ray described the talk as being a bit “Neo-Piagetian theory for dummies” (for which I am eternally grateful)  and was seeking to address the question as to where students are actually operating when we ask them to undertake tasks that require a reasonable to high level of intellectual development.

Ray raised the three bad programming habits he’d discussed earlier:

  1. Permutation programming (where students just try small things randomly and iteratively in the hope that they will finally get the right solution – this is incredibly troublesome if the many small changes take you further away from the solution )
  2. Shotgun debugging (where a bug causes the student to put things in with no systematic approach and potentially fixing things by accident)
  3. Voodoo coding/Cargo cult coding (where code is added by ritual rather than by understanding)

These approaches show one very important thing: the student doesn’t understand what they’re doing. Why is this? Using a Neo-Piagetian framework we consider the student as moving through the same cognitive development stages that they did as a child (Piagetian) but that this transitional approach applies to new and significant knowledge frameworks, such as learning to program. Until they reach the concrete operational stage of their development, they will be applying poor or inconsistent models – logically inadequate models to use the terminology of the area (assuming that they’ve reached the pre-operational stage). Once a student has made the next step in their development, they will reach the concrete operational stage, characterised (among other things, but these were the ones that Ray mentioned) by:

  1. Transitivity: being able to recognise how things are organised if you can impose an order upon them.
  2. Reversibility: that we can reverse changes that we can impose.
  3. Conservation: realising that the numbers of things stay the same no matter how we organise them.

In coding terms, these can be interpreted in several ways but the conservation idea is crucial to programming because understanding this frees the student from having to write the same code for the same algorithm every time. Grasping that conversation exists, and understanding it, means that you can alter the code without changing the algorithm that it implements – while achieving some other desirable result such as speeding the code up or moving to a different paradigm.

Ray’s paper discussed the fact that a vast number of our students are still pre-operational for most of first and second year, which changes the way that we actually try to teach coding. If a student can’t understand what we’re talking about or has to resort to magical thinking to solve problem, then we’ve not really achieved our goals. If we do start classifying the programming tasks that we ask students to achieve by the developmental stages that we’re expecting, we may be able to match task to ability, making everyone happy(er).

The final paper in the session was “Social Sensitivity Correlations with the Effectiveness of team Process Performance: An Empirical Study”, (Luisa Bender (presenting), Gursimra Walia, Krishna Kambhampaty, Travis Nygard and Kendall Nygard), which discussed the impact of socially sensitive team members in programming teams. (Social sensitivity is the ability to correctly understand the feelings and the viewpoints of other people.)

The “soft skills” are essential to teamwork process and a successful team enhances learning outcomes. Bad teams hinder team formation and progress, and things go downhill from there. From Wooley et al’s study of nearly 700 participants, the collective intelligence of the team stems from how well the team works rather than the individual intelligence of the participants. The group whose members were more socially sensitive had a higher group intelligence.

Just to emphasise that point: a team of smart people may not be as effective as a team as a team of people who can understand the feelings and perspectives of each other. (This may explain a lot!)

Social sensitivity is a good predictor of team performance and the effectiveness of team-oriented processes, as well as the satisfaction of the team members. However, it is also apparent that we in Science, Technology, Engineering and Mathematics (STEM) have lower social sensitivity readings (supporting Baron-Cohen’s assertion – no, not that one) than some other areas. Future work in this area is looking at the impact of a single high or low socially sensitive person in a group, a study that will be of great interest to anyone who is running teams made up on randomly assigned students. How can we construct these groups for the best results for the students?

More MOOCs! (Still writing up ICER, sorry!)

Posted: September 12, 2012 | Author: | Filed under: Education | Tags: , , , , , , , , , , | Leave a comment

The Gates Foundation is offering grants for MOOCs in Introductory Classes. I mentioned in an earlier post that if we can show that MOOCs work, then generally available and cheap teaching delivery is a fantastically transformative technology. You can read the press release but it’s obvious that this has some key research questions in it, much as we’ve all been raising:

The foundation wants to know, for instance, which students benefit most from MOOC’s (sic) and which kinds of courses translate best to that format.

Yes! If these courses do work then for whom do they work and which courses? There’s little doubt that the Gates have been doing some amazing things with their money and this looks promising – of course, now I have to find out if my University has been invited to join and, if so, how I can get involved. (Of course, if they haven’t, then it’s time to put on my dancing trousers and try to remedy that situation.)

However, money plus research questions is a good direction to go in.

A side post on MOOCs: angrymath Hates Statistics 101

Posted: September 11, 2012 | Author: | Filed under: Education | Tags: , , , , , , , , | Leave a comment

A friend just forwarded me a rather scathing critique of one of the Udacity courses. The rather aptly named angrymath has published Udacity Statistics 101. To forewarn you, this is one of the leading quotes:

In brief, here is my overall assessment: the course is amazingly, shockingly awful.

As one of the commenters put it, hopefully the problems are growing pains and iteration towards perfection will continue. I haven’t seen the course in question so can’t comment, merely present.

ICER 2012 Day 1: Discussion Papers Session 1

Posted: September 11, 2012 | Author: | Filed under: Education | Tags: , , , , , , , , , , , , | Leave a comment

ICER contains a variety of sessions: research papers, discussion papers, lightning talks and elevator pitches. The discussion papers allow people to present ideas and early work in order to get the feedback of the community. This is a very vocal community so opening yourself up to discussion is going to be a bit like drinking from the firehouse: sometimes you quench your thirst for knowledge and sometimes you’re being water-cannonned.

Web-scale Data Gathering with BlueJ
Ian Utting, Neil Brown, Michael Kölling, Davin McCall and Philip Stevens

BlueJ is a very long-lived and widely used Java programming environment with a development environment designed to assist with the learning and teaching of object-oriented programming, as well as Java. The BlueJ project is now adding automated instrumentation to every single BlueJ installation and students can opt-in to a data reporting mechanism that will allow the collection and formation of a giant data repository: Project Blackbox. (As a note, that’s a bit of a super villain name, guys.)

BlueJ has 1-2M New users per year, typically using it for ~90 days and all of these users will be able to opt-in, can opt-out later, although this can be disabled in config. To protect user identity, locally generated (anon) UUID will be generated and linked to user+installation (So home and lab won’t correlate). On the technical side, the stored data will includes time-stamps, tool invocations, source code snapshots, and course-grained location. You can also connect (locally available) personal data about students and link it to UUID data. Groups can be tagged and queries restricted to that tag (and that includes taxonomic data if you’re looking into the murky world of assessment taxonomy).
In terms of making this work, ethical approval has been obtained from the hosting organisation, for verified academic researchers, initially via SQL queries on multi-terabyte repository but the data will not be fully public (this will be one of largest repositories of assignment solutions in the world).
Timescale: private beta by end of 2012, with a full-scale roll out next Spring, AY 2013. Very usefully, you can still get access to the data even if you don’t contribute.
There was a lot of discussion on this: we’re all hungry for the data. One question that struck me was from Sally Fincher: Given that we will have web-scale data gathering, do we have web scale questions? We can all think of things to do but this level of data is now open to entirely new analyses. How will we use this? What else do we need to do?

Evaluating an Early Software Engineering Course with Projects and Tools from Open Source Software
Robert McCartney, Swapna Gokhale and Therese Smith

We tend to give Software Engineering students a project that requires them to undertake design and then, as a group, produce a large software artefact from scratch. In this talk, Robert discussed using existing projects that use a range of skills that are directly relevant to one of the most common activities our students will carray out in industry: maintenance and evolution.

Under a model of developing new features in an open-source system, the instructors provide a pre-selected set of projects and then the 2 person team:

  1. picks a project
  2. learns to comprehend code
  3. proposes enhancements
  4. describes and documents
  5. implements and presents
The evaluation seeks to understand how the students’ understanding of issues has changed especially regarding the importance of maintenance and evolution, the value of documentation, the importance of tools and how reverse engineering can aid comprehension. This approach has been trialled and early student response is positive but the students thought that 10,000 Lines of Code (LOC) projects were too small, hence the project size has increased to 100,000 LOC.

A Case Study of Environmental Factors Influencing Teaching Assistant Job Satisfaction
Elizabeth Patitsas

Elizabeth presented some interesting work on the impact of lecture theatres on what our TAs do. If the layout is hard to work with then, unsurprisingly, the TAs are less inclined to walk around and more inclined to disengage, sitting down the front checking e-mail. When we say ‘less inclined’, we mean that in closed lab layouts TAs spend 40% of the their time interacting with students, versus 76% in an open layout. However, these effects are also seen in windowless spaces: make a space unpleasant and you reduce the time that people spend answering questions and engaging.

The value of a pair of TAs was stressed: a pair gives you a backup but doesn’t lead to decision problems when coming to consensus. However, the importance of training was also stressed, as already clearly identified in the literature.

Education and Research: Evidence of a Dual Life
Joe Mirõ Julia, David López and Ricardo Alberich

Joe provided a fascinating coloration network analysis of the paper writing groups in ICER and generally. In CS education,  we tend to work in smaller groups than other CS research areas and newcomers tend to come alone to conferences. The ICER colouration network graph has a very well-defined giant component that centres around Robert (see above) but, across the board, roughly 50% of conference authors are newcomer. One of the most common ways for people to enter the traditional CS research community is through what can be described as a mentoring process, we extend the group through an existing connection and then these people join the giant component. There is, however, no significant evidence of mentoring in the edu community.
Unsurprisingly, different countries and borders hinder the growth of the giant component.
There was a lot of discussion on this as well, as we tried to understand what was going on and, outside of the talk, I raised my suggestion with Joe that hemispherical separation was a factor worth considering because of the different timetables that we worked to. Right now, I am at a conference in the middle of teaching, while the Northern Hemisphere has only just gone back to school.

ICER 2012 General Note

Posted: September 11, 2012 | Author: | Filed under: Education | Tags: , , , , , | Leave a comment

Once again, we’re so full of interesting content that I don’t really have the time to put together some longer posts although I’m going to try and get something out over the tea break and lunch. In short, if you get a chance, COME TO ICER.

I will however note, while I can transcribe a lot of speakers almost as fast as they can deliver interesting talks, my top speed is asymptotically bound at an upper limit that I am officially designating One Guzdial.

ICER 2012 Day 1 Keynote: How Are We Thinking?

Posted: September 10, 2012 | Author: | Filed under: Education | Tags: , , , , , , , , , , , , , , , | 3 Comments

We started off today with a keynote address from Ed Meyer, from University of Queensland, on the Threshold Concepts Framework (Also Pedagogy, and Student Learning). I am, regrettably, not as conversant with threshold concepts as I should be, so I’ll try not to embarrass myself too badly. Threshold concepts are central to the mastery of a given subject and are characterised by some key features (Meyer and Land):

  1. Grasping a threshold concept is transformative because it changes the way that we think about something. These concepts become part of who we are.
  2. Once you’ve learned the concept, you are very unlikely to forget it – it is irreversible.
  3. This new concept allows you to make new connections and allows you to link together things that you previously didn’t realise were linked.
  4. This new concept has boundaries – they have an area over which they apply. You need to be able to question within the area to work out where it applies. (Ultimately, this may identify areas between schools of thought in an area.)
  5. Threshold concepts are ‘troublesome knowledge’. This knowledge can be counter-intuitive, even alien and will make no sense to people until they grasp the new concept. This is one of the key problems with discussing these concepts with people – they will wish to apply their intuitive understanding and fighting this tendency may take some considerable effort.

Meyer then discussed how we see with new eyes after we integrate these concepts. It can be argued that concepts such as these give us a new way of seeing that, because of inter-individual differences, students will experience in varying degrees as transformative, integrative, and (look out) provocative and troublesome. For this final one, a student experiences this in many ways: the world doesn’t work as I think it should! I feel lost! Helpless! Angry! Why are you doing this to me?

How do you introduce a student to one of these troublesome concepts and, more importantly, how can you describe what you are going to talk about when the concept itself is alien: what do you put in the course description given that you know that the student is not yet ready to assimilate the concept?

Meyer raised a really good point: how do we get someone to think inside the discipline? Do they understand the concept? Yes. Does this mean that they think along the right lines? Maybe, maybe not. If I don’t think like a Computer Scientist, I may not understand why a CS person sees a certain issue as a problem. We have plenty of evidence that people who haven’t dealt with the threshold concepts in CS Education find it alien to contemplate that the lecture is not the be-all and end-all of teaching – their resistance and reliance upon folk pedagogies is evidence of this wrestling with troublesome knowledge.

A great deal to think about from this talk, especially in dealing with key aspects of CS Ed as the threshold concept that is causing many of our non-educational research oriented colleagues so much trouble, as well as our students.

 

ICER 2012: So Good I Don’t Have Time To Blog!

Posted: September 10, 2012 | Author: | Filed under: Education | Tags: , , , , | Leave a comment

I’m going to try and post when I can but the conference is so good that there’s nothing I can skip. Apologies, I shall try and dump my notes from today when I have a chance!

ICER 2012: Day 0 (Workshops)

Posted: September 10, 2012 | Author: | Filed under: Education | Tags: , , , , , , , , , , , , , , , , , | 1 Comment

Well, it’s Sunday so it must be New Zealand (or at least it was Sunday yesterday). I attended that rarest of workshops, one where every session was interesting and made me think – a very good sign for the conference to come.

We started with an on-line workshop on Bloom’s taxonomy, classifying exam questions, with Raymond Lister from UTS. One of the best things about this for me was the discussion about the questions where we disagreed: is this application or synthesis? It really made me think about how I write my examinations and how they could be read.

We then segued into a fascinating discussion of neo-Piagetian theory, where we see the development stages that we usually associate with children in adults as they learn new areas of knowledge. In (very rough) detail, we look at whether we have enough working memory to carry out a task and, if not, weird things happen.

Students can indulge in some weird behaviours when they don’t understand what’s going on. For example, permutation programming, where they just type semi-randomly until their program compiles or works. Other examples include shotgun debugging and voodoo programming and what these amount to are the student not having a good consistent model of what works and, as a result, they are basically dabbling in a semi-magic approach.

My notes from the session contain this following excerpt:

“Bizarro” novice programmer behaviours are actually normal stages of intellectual development.
Accept this and then work with this to find ways of moving students from pre-op, to concrete op, to formal operational. Don’t forget the evaluation. Must scaffold this process!

What this translates to is that the strange things we see are just indications that students having moved to what we would normally associate with an ‘adult’ (formal operational) understanding of the area. This shoots several holes in the old “You’re born a programmer” fallacy. Those students who are more able early may just have moved through the stages more quickly.

There was also an amount of derisive description of folk pedagogy, those theories that arise during pontification in the tea room, with no basis in educational theory or formed from a truly empirical study. Yet these folk pedagogies are very hard to shake and are one of the most frustrating things to deal with if you are in educational research. One “I don’t think so” can apparently ignore the 70 years since Dewey called the classrooms prisons.

The worst thought is that, if we’re not trying to help the students to transition, then maybe the transition to concrete operation is happening despite us instead of because of us, which is a sobering thought.

I thought that Ray Lister finished the session with really good thought regarding why students struggle sometimes:

The problem is not a student’s swimming skill, it’s the strength of the torrent.

As I’ve said before, making hard things easier to understand is part of the job of the educator. Anyone will fail, regardless of their ability, if we make it hard enough for them.

Post 300 – 2012, the Year of the Plague

Posted: September 9, 2012 | Author: | Filed under: Education, Opinion | Tags: , , , , , , , , , | Leave a comment

As it turns out, this is post 300 and I’m going to use it to make a far more opinionated point than usual. I’m currently in Auckland, New Zealand, and there is a warning up on the wall about a severe outbreak of measles. This is one of the most outrageously stupid signs to see on a wall, anywhere, given that we have had a solid vaccine since 1971 and, despite ill-informed and unscientific studies that try to contradict this, the overall impact of the MMR vaccine is overwhelmingly positive. There is no reasonable excuse for the outbreak of an infectious, dangerous disease 40 years after the development of a reliable (and overwhelmingly safe) vaccine.

Is this really what we want?

My fear is that, rather than celebrating the elimination of measles and polio (under 200 cases this year so far according to the records I’ve seen) in the same way that we eradicated smallpox, we will be seeing more and more of these signs identifying outbreaks of eradicable and controllable diseases, because ignorance is holding sway.

Be in no doubt, if we keep going down this path, the risk increases rapidly that a disease will finish us off because we will not have the correct mental framing and scientific support to quickly respond to a lethal outbreak or mutation. The risk we take is that, one day, our cities lie empty with signs like this up all over the place, doors sealed with crosses on them, a quiet end to a considerable civilisation. All attributable to a rejection of solid scientific evidence and the triumph of ignorance. We have survived massive outbreaks before, even those with high lethality, but we have been, for want of a better word, lucky. We live in much denser environments and are far more connected than we were before. I can step around the world in a day and, with every step, a disease can follow my footsteps.

One of my students recently plotted 2009 Flu cases relative to air routes. While disease used to rely upon true geographical contiguity, we now connect the world with the false adjacency of the air route. Outbreaks in isolated parts of the world map beautifully to the air hubs and their importance and utilisation: more people, higher disease.

So, in short, it’s not just the way that we control the controllable diseases that is important, it is accepting that the lower risk of vaccination is justifiable in the light of the much greater risk of infection and pandemic. This fights the human tendency to completely misunderstand probability, our susceptibility to fallacious thinking, and our desperate desire to do no harm to our children. I get this but we have to be a little bit smarter or we are putting ourselves at a much higher risk – regrettably, this is a future risk so temporal discounting gets thrown into the mix to make it ever harder for people to make a good decision.

Here’s what the Smallpox Wikipedia page says: “Smallpox was an infectious disease unique to humans” (emphasis mine). This is one of the most amazing things that we have achieved. Let’s do it again!

I talk a lot about education, in terms of my thoughts on learning and teaching, but we must never forget why we educate. It’s to enlighten, to inform, to allow us to direct our considerable resources to solving the considerable problems that beset us. It’s helping people to make good decisions. It’s being aware of why people find it so hard to accept scientific evidence: because they’re scared, because someone lied to them, because no-one has gone to the trouble to actually try and explain it to them properly. Ignorance of a subject is the state that we occupy before we become informed and knowledgable. It’s not a permanent state!

That sign made me angry. But it underlined the importance of what it is that we do.

← Older Entries
Newer Entries →