Dr Falkner Goes to Canberra Day 1 “Opposition Leader Address” (#smp2014 #AdelED @billshortenmp)

Posted: March 17, 2014 | Author: | Filed under: Education | Tags: , , , , , , , , | Leave a comment

The start of the Opposition Leader’s Address we delayed because he was doing media outside (chuckles from the crowd). While we waited, Simon took Q&A. The first question was “How do we get a better media for reporting science/more accountable media?” This was thrown to the Australian Science and Media association but briefings are apparently key. Ah, here’s the Opposition Leader, The Hon Bill Shorten MP, Leader of the Opposition (@billshortenmp). Probably the last thing he really wants to do given the recent elections but he’s looking pretty chipper. (Strong applause from the crowd)

Mr Shorten referred to the dark arts of politics and the media, as Labour leader and as science policy leader for his party. He encouraged us to give 1% of ourselves to political communicators – stay 99% science but get ready to explain things to the people on the hill and Australians. Science and Innovation are matters of national political importance and what scientists to is important to the future of Australian in 10-30 years in the future. A reference was made to shunting of science into DoI with no dedicated Science minster (here heres, from the crowd). Industry needs science. Health needs new ideas. Education needs new ideas. The economy needs new ideas underpinned by innovation, research and education. Basically, Australia needs science. Environmental policy should be based on scientific consensus rather than ideological repudiation (his words).

The beauty and rightness of science will not guarantee its success – it has to be communicate properly to become a successful political issue. That’s a bitter pill but it’s probably the right medicine. Mr Shorten referred to Einstein’s musing on politics, that politics is more difficult than physics. He referred to disability issues that, prior to 2007-8, were not regarded as a national political issue – it was a moral imperative but, until a national disability initiative, it didn’t feature on the national level in a meaningful and contributory way. (I’m paraphrasing here without editorial, you may disagree with Mr Shorten’s perspective.) The two key elements that moved it from charity to outcome, from forgotten tragedy to an issue supported in a bipartisan fashion. The first factor was people who knew about the issue raising the profile with a positive-message focused grass roots message. Different groups on the same issue can either work together or form a Tower of Babel. Are you focusing on the 10% of things you disagree on or the 90% you agree on? The sterile competition of conflicting points of view fighting for the same resources are counter-productive. The groups then focussed on a single successful outcome – the national disability insurance scheme. We are all fighting for limited resources – but we can still have a unifying message. A consensus that will delver goals that will benefit all of us and build a richer political narrative of the benefits of science for all Australians.

The second factor was support evidence from recognised experts who could provide a cogent argument as to how this approach will benefit all Australians, not just parliamentarians. The productivity commission were able to puncture the myth that a national disability insurance scheme would be a bottomless pit of debt – this made the policy more attractive. As well, the productivity commission argued that empowering all of the people associated with the disabled would be a net positive. This added economic soundness and policy logic to the moral imperative of the initiative.

Innovative Australian business are 78% more likely to report improvements in productivity but only 25% of Australian businesses collaborate on this innovation. There is a great need for more research collaboration. Industry relies on science and we, as scientists, have a role in communicating and collaborating as part of this.

Building meaningful consensus in parliament is not easy and it’s harder for science. Simply presenting the same system of funding and expecting a different outcome would be a pipe dream. Science is under constant assault from the crank blogosphere and fringe opinion is often presented in the media as being an alternative – which is irritating, as Mr Shorten noted. We need to triumph over gossip and prejudice. All good stuff, but we need to be able to communicate what we do, even if it means stepping back to the first principles.

Mr Shorten spoken on climate change and noted that we often make the mistake of assuming that our view of a community, especially within the scientific community, can be mapped to the general community, which means that we get caught flat-footed when the people are swayed by poor, incorrect, misleading and negative reporting. (We’ve seen the bubble before in the Bush/Gore election.) Mr Shorten, it would fair to say, disagrees with the government on this. None of us were shocked by this.

Mr Shorten called for a discussion of real science, evidence-based research, and far more informed work in parliament, including letting projects run to completion without ideological interference and, of great interest to me, to also being able to change direction when we discover that we are on the wrong path – but without being mocked for so-called indecisiveness. He sees Science as the industry that will underwrite our successes in the 21st Century.

We are more than just a rock or a crop. The Hon Bill Shorten, MP, Leader of the Opposition.

Science needs a long term and sustainable funding profile and we need to focused on educating more scientists now, because too many students have no real grasp of science because we have neither the teachers nor the desire to increase the knowledge. Australia can either get smarter or poorer, compete or give up. Then there were some more partisan points but I was still interested to hear about allowing failure and recognising that failure sometimes is a key part of the movement to success. Our scientist graduates should have professional skills as well as their discipline skills (something I also agree with, I note), but they should also have good lives where they don’t have to flog themselves to get ahead.

Politics might be a dirty business but we can’t stay hands off any more – giving 1% of your science brain to working out how to communicate and enter the political debate in a way that makes change happen.

(Nick: Very interesting talk indeed. Not a great week for Labour but it was a relatively simple and powerful message: get out of your lab and push your message further.)

The first question was on the nuclear fuel cycle and how it related to ALP policy, with a plea for it to be treated fairly on its scientific and environmental merits. The second question was on what message do we need to give Young Scientists to get engaged (the flippant answer he gave was “Join the Labor party”, which he modified to “Join any party, but you’ll be less happy elsewhere.”)

The answer to the first question summarised Labor policy and focused on the cost of starting a new technology cycle now but he conceded that the debate should be held on its merits and noted that the Far Left approach could be as fundamentalist as the Far Right. He then raised storage and economic start-up issues. (I shall wait for arguments in the comments. 🙂 )

The second question was “get political”, even to the point of going into parliament. Politics are not that mysterious and scientists need to believe that politics can change the community and can speak to the lives of everyday Australians. (First reference to rebuilding Labor.)

Dr Falkner Goes to Canberra Day 1 “Inspiring Australia: how can we help you communicate your science?” (#smp2014 #AdelED #questacon @questacon)

Posted: March 17, 2014 | Author: | Filed under: Education | Tags: , , , , , | Leave a comment

questacon-logo_0.png

The speaker is Simon France, Program Manager, Questacon (@questacon). Simon had recently returned from Chicago and was noting that we had here in terms of science engagement is unique and desirable from another country’s perspective, in terms of our strategy. If we were confident that parliamentarians were listening, science literate and, even when they differed, we knew that they were taking it seriously – that would be fantastic. It would be great if your neighbours were also taking science seriously. It would be great if every graduating student in Australia, whether in Science or not, understood enough science to take part in serious and well-fromed debate. This is part of the Inspiring Australia initiative. This, of course, requires scientists to be able to communicate in a way that facilitates this, another part of Inspiring Australia.

Simon then showed a short video talking about how we can match the interest of an audience to the message of a scientist – which led to the formation of the Inspiring Australia initiative. The groups formed as part of this is designed to provide leadership in scientific communications, supported by experts in the area, to work with media, evidence-based research, and indigenous strategic elements.

What can this offer us? Media training skills – are we able to work with media as we want to? What about social media? (I’ve probably got that covered but let’s see about that.) There’s a … three week … social media training program, as well as a science in newsroom program. (Three weeks???) We have to communicate science more effectively and Questacon have platforms and training – National Science Week (which I’ve done before) is one of these activities, with 1.6 million Australians involved. A number of other initiatives were discussed, including national hubs for regional science promotion, on the collective side. One group, the Science Sector Group, is trying to build up a strong message on one theme to drive change.  Questacon has a booth here and they want to know where to go next – I’ll have to think about that before morning tea! What training and tools do we need? How can we engage better with industry? How can we do citizen science better? Prizes? Clubs? Science and tourism? How do we communicate the value and importance of science in Australia?

When we communicate poorly, we might not get our real message across. What do our listeners miss when we confuse and mumble our message? (I have a lot more to say on this but will have to update after this event.) Have you tested your talk on someone out of your area or who is more critical before you present? Interesting…

Dr Falkner Goes to Canberra Day 1 “Welcome and Overview” (#smp2014 #AdelED)

Posted: March 17, 2014 | Author: | Filed under: Education | Tags: , , , , , , | Leave a comment

Web-Banner11

(Disturbing social media fact: the people sitting next to me had seen my breakfast view tweet on the Canberra twitter site before they met me. I think I’m having my 15 bits of fame.)

(Colour aside: there’s a lot of orange in use here. I feel at home.)

The welcome and overview to the session was given by Dr Ross Smith (President) and Catriona Jackson (CEO) of Science and Technology Australia (STA). This is the 29th year of the even, which is a pretty impressive run! This event is being supported by the Department of Industry and a host of sponsors (who I won’t list here although I will note that a couple of Unis were sponsoring, which I found interesting, along with the NTEU). This has attracted a lot of sponsorship. There are roughly 200 scientists here, for development activities to improve our ability to communicate with parliament and the media.

Science thrives in the creation of networks.

There’ll be a launch of the Parliamentary Friendship Group for Science tonight, which sounds very interesting and I’m looking forward to find out what that’s about. We have almost half the whole parliament arranged for meetings tomorrow – about 120 people – which is quite surprising and a very high level of exposure to parliament. There’s a media minder available, which I doubt I’ll need as there’s still a very low level of media interest in much of the educational stuff I work with, but it’s good to know that they’re there.

Today we’re in the NGA but tomorrow we’re in the “belly of the parliament” and have very strict rules for when we can enter the parliament. I suspect I’ll be having a very early morning tomorrow to make the early breakfast entry point. (I must check to see if there are equipment limits to what we can bring in, given I travel with a small electronics shop.)

That’s it, time for the first talk!

Dr Falkner Goes to Canberra – Day 1 (#smp2014 #AdelED #CORE)

Posted: March 17, 2014 | Author: | Filed under: Education | Tags: , , , , , , , , , | Leave a comment

Settling in to Canberra after a relaxing night in the hotel working (hooray for high-speed WiFi and good desks) and then hammering myself for an hour in the gym to make sure that I was really awake and ready for a power breakfast. (That’s secret Nick code for ‘bacon’)

The starting venue for this year’s Science Meets Parliament is the National Gallery of Australia, Gandel Hall. I’m here representing CORE – the Computing Research and Education association. CORE gets two seats at this event and I was lucky enough to secure one. The focus of this whole activity is to show scientists how Parliament actually works and, with any luck, how to engage successfully with this level of organisation – but also to get Parliamentarians and scientists talking together. It’s a pretty high profile line-up and my own meeting looks like it will be pretty interesting.

The two days involve a lot of briefings, discussions and meeting opportunities as well as the chance to attend the Press Club luncheon tomorrow (which, sadly, I won’t be able to attend as my meeting with a Parliamentarian conflicts with this) and sitting in on question time.

Right now the room is slowly filling up as people register and find tables. Demographically, there are roughly 40% women, and very few non-white faces. The overall age is not that great (or grey, if I may), which isn’t much of a surprise.

We start in about 30 minutes so I’ll resume blogging then.

ASWEC 2014 – Now with Education Track

Posted: March 12, 2014 | Author: | Filed under: Education | Tags: , , , , , , , , , , , , , , , , | Leave a comment

The Australasian Software Engineering Conference has been around for 23 years and, while there have been previous efforts to add more focus on education, this year we’re very pleased to have a full day on Education on Wednesday, the 9th of April. (Full disclosure: I’m the Chair of the program committee for the Education track. This is self-advertising of a sort.) The speakers include a number of exciting software engineering education researchers and practitioners, including Dr Claudia Szabo, who recently won the SIGCSE Best Paper Award for a paper in software engineering and student projects.

Here’s the invitation from the conference chair, Professor Alan Fekete – please pass this on as far as you can!:

We invite all members of the Australasian computing community to ASWEC2014, the Australasian Software Engineering Conference, which will be held at VIBE hotel in Milson’s Point, Sydney, on April 7-10, 2014.
 
The conference program is at http://www.aswec2014.org/programme/schedule/ Highlights include
  • Keynote by a leader of SE research, Prof Gail Murphy (UBC, Canada) on Getting to Flow in Software Development.
  • Keynote by Alan Noble (Google) on Innovation at Google.
  • Sessions on Testing, Software Ecosystems, Requirements, Architecture, Tools, etc, with speakers from around Australia and overseas, from universities and industry, that bring a wide range of perspectives on software development.
  • An entire day  (Wed April 9) focused on SE Education, including keynote by Jean-Michel Lemieux (Atlassian) on Teaching Gap: Where’s the Product Gene?
Register at http://www.aswec2014.org/registration/ Single-day registration is available. The conference is colocated with WICSA http://www.wicsa.net/  (International Conference on Software Architecture) and immediately preceded by an SEI course on software architecture http://www.australia.cmu.edu/events/software-architecture-principles-and-practices
We look forward to seeing many of you at ASWEC2014!

 

SIGCSE Day 3, “MOOCs”, Saturday, 10:45-12:00pm, (#SIGCSE2014)

Posted: March 9, 2014 | Author: | Filed under: Education | Tags: , , , , , , , , , , , , , , , , , , | Leave a comment

This session is (unsurprisingly) of very great interest to the Adelaide Computer Science Education Research group and, as the expeditionary force of CSER, I’ve been looking forward to attending. (I’d call myself an ambassador except I’m not carrying my tuxedo.) The opening talk was “Facilitating Human Interaction in an Online Programming Course” presented by Joe Warren, from Rice University. They’ve been teaching a MOOC for a while and they had some observations to share on how to make things work better. The MOOC is an introduction to interactive programming in Python, based one that Joe had taught for years, which was based on building games. First on-line session was in Fall, 2012, after  face-to-face test run. 19,000 students completed three offerings over Fall ’12, Spring ’13 and Fall ’13.

The goal was to see how well they could put together a high quality on-line course. They sussed recorded videos and machine-graded quizzes, with discussion forums and peer-assessed mini projects. They provided a help desk manned by course staff. CodeSkulptor was the key tool to enable human interaction, a browser based IDE for Python, which was easy to set up, and cloud-saved URLs for code, which were easy to share. (It’s difficult to have novices install tools without causing problems and code visibility is crucial for sharing.) Because they needed a locally run version of Python for interactivity (games focus) so they used Skulpt which translated Python into JavaScript, combined it with CodeMirror, an editor, and then ran it in the browser. CodeSkulptor was built on top.

Students could write code and compile it in the browser, but when they save it a hash is generated for unique storage in a cloud-based account with an access URL – anyone can run your code if you share the URL. (The URL includes a link to CodeSkulptor.org) CodeSkulptor has about 12 million visits with 4 million files saved, which is pretty good. The demo shown had keyboard input, graphic images and sound output – and for those of you know about these things, this is a great result without having to install a local compiler – the browser-based solution works pretty well.

Peer assessment occurred at weekly mini-projects, where the Coursera course provided URLs for CodeSkulptor and a grading rubric which gets sent to students in a web-form. The system isn’t anonymised but students knew it was shared and were encouraged to leave out any personal details in their comments if they wanted to be anonymous (as the file handles were anonymised). (Apparently, the bigger problem was inappropriate content, rather than people worrying about anonymity.) The students run it, assess it in about 10 minutes so it takes about an hour to assess 6 peers. The big advantage is that the code form your URL is guaranteed to run on the grader’s machine because it’s the same browser-based environment. A very detailed rubric was required to ensure good grading: lots of small score items with little ambiguity. The rubric did’t leave much room for assessment – the students were human machines. Why? Having humans grade it was an educational experience and learned from reading and looking at each other’s programs. Also, machine graders have difficulty with animated games, so this is a generalisable approach.

The Help Desk addressed the problem of getting timely expert help for the students – this is a big issue for students. The Code Clinic had custom e-mail that focuses on coding problems (because posting code was not allowed under the class Honour Code). Solutions for common problems were then shared to the rest of the class via the forum. (It looked like the code hash changed every time it got saved? That is a little odd from a naming perspective if true.)

How do CodeClinic work? in Spring 2013 they had about 2,500 help requests. On due days, response time was about 15 minutes (usually averaged 40+), overall handling time average was 6 minutes (open e-mail, solve problem respond). Over 70 days, 3-4 staff sent about 4,000 e-mails. That handling time for a student coding request is very short and it’s a good approach to handling problems at scale. That whole issue about response time going DOWN on due date days is important – that’s normally where I get slammed and slow down! It’s the most popular class at the Uni, which is great!

The chose substantial human-human interaction, using traditional methods on line with peer assessment nd help desks. MOOCs have some advantages over in-person classes – the forums are active because of their size and the help desk scaling works really effectively because it’s always used and hence it makes sense to always staff it. The takeaway is that you have to choose your tools well and you’ll be able to do some good things.

The second talk was “An Environment for Learning Interactive Programming”, also from Rice, and presented by Terry Tang. There was a bit of adverblurb at the start but Terry was radiating energy so I can’t really blame him. He was looking at the same course as mentioned in the previous talk (which saves me some typing, thank you session organisers!). In this talk, Terry was going to focus on SimpleGUI, a browser-based Python GUI library, and Viz mode, a program visualisation tool. (A GUI is a  Graphical User Interface. When you use shapes and windows to interact with a computer, that’s the GUI.)

Writing games requires a fully-functional GUI library so, given the course is about games, this had to be addressed! One could use an existing Python library but these are rarely designed to support Python in the browser and many of them are too complicated as APIs for novice programmers (good to see this acknowledged!).  Desired features of the new library: event-driven support, drawing support and enable students to be able to create simple but interesting programs. So they wrote SimpleGUI . Terry presented a number of examples of this and you can read about it in the talk. (Nick code for “I’m not writing that bit.”) The program was only 227 lines long because a lot of the tricky stuff was being done in the GUI.

Terry showed some examples of student code, built from scratch, on the SimpleGUI, and showed us a FlappyBirds clone – scoring 3, which got a laugh from the crowd.

Terry then moved on to Viz mode, to meet the requirement of letting students visualise the execution of their own code. One existing solution is the Online Python Tutor, which runs code on a server, generates a log file and then ships the trace to some visualisation code in the browse (in JavaScript) to process the trace and produce a state diagram. The code, with annotations, is presented back to the user and they can step through the code, with the visualisations showing the evolution of state and control over time. The resulting visualisation is pretty good and very easy to follow. Now this is great but it runs on a backend server, which could melt on due dates, and OPT can’t visualise event-driven programs (for those who don’t know, game programming is MOSTLY event-driven). So they wrote Viz mode.

From CodeSkulptor, you can run your program in regular or Viz mode. In Viz mode, a new panel with state diagrams shows up, and a console that shows end tags. This is all happening in the browser, which scales well although there are limits to computation in this environment, and is all integrated with the existing CodeSkulptor environment. Terry then showed some more examples.

An important note is that event handlers don’t automatically fire in Viz mode so an GUI elements will have additional buttons to explicitly fire events (like Draw for graphical panes or Timers for events like that). It’s a pretty good tool, from what we were shown. Overall, the Rice experience looks very positive but their tool set and approach to support appears to be the keys to their success. Only some of the code is open source, which is a pity.

Barb Ericson asked a great question: could you set up something where the students are forced to stop and then guess what is going to happen next? They haven’t done it yet but, as Joe said, they might do it now!

The final talk was not from Rice but was from Australia, woohoo (Melbourne and NICTA)! “Teaching Creative Problem Solving in a MOOC” was presented by Carleton Coffrin from NICTA. Carleton was at Learning@SCale earlier and what has been seen over the past year is MOOCs 1.0 – scaling content delivery, with linear delivery, multiple-choice questions and socialisation only in the forums. What is MOOC 2.0? Flexible delivery, specific assessments, gamification, pedagogy, personalised and adaptive approaches. Well, it turns out that they’ve done it so let’s talk about it with a discrete optimisation MOOC offered on Coursera by University of Melbourne. Carleton then explained what discrete optimisation was – left to you to research in detail, dear reader, but it’s hard and the problems are very complex (NP-hard for those who care about such things). Discrete optimisation in practice is trying to explain known techniques to complicated real-world problems. Adaptation and modification of existing skills is a challenge.

How do we prepare students for new optimisation problems that we can’t anticipate? By teaching general problem-solving skills.

What was the class design? The scope of the course was over six areas in the domain, which you can find in the paper, and five assignments of NP-hard complexity. In the lectures, the lecturer used a weatherman format with a lecturer projected over the slides with a great deal of enthusiasm – and a hat. (The research question of the global optimum for hats was not addressed.) The lecturer was very engaging and highly animated, which added to the appeal of the recorded lectures. The instructor constructs problems, students write code and generate solution, encode the solution in a standard format, this is passed back, graded and feedback is returned. Students get the feedback and can then resubmit until the student is happy with their grade. (Yay! I love to see this kind of thing.) I will note that the feedback told them what quality of solution that had to present rather than suggestions of how to do it. Where constraint violations occurred, there was some targeted feedback. Overall, the feedback was pretty reasonable but what you’d expect in good automated feedback. The students did demonstrate persistence in response to this feedback.

From a pedagogical perspective, discovery-based learning was seen to be very important as part of the course. Rather than teach mass, volume and density by using a naked formula, exemplars were presented using water and floating (or sinking) objects to allow the students to explore the solutions and the factors. The material is all in the lectures but it’s left to the students to find the right approach to find solutions to new problems – they can try different lecture ideas on different problems.

The instructor can see all of the student results, rank them, strip out the results and then present a leader board to show quality. This does allow students to see that higher numbers are achieved but I’m not sure that there’s any benefit beyond what’s given in the hints. They did add a distribution graph for really large courses as the leader board got too long. (I’m not a big fans of leader boards, but you know that.)

The structure of the course was suggested, with introductory materials, but then students could bounce around. On-line doesn’t require a linear structure! The open framework was effectively require for iterative improvement of the assignments.

How well did it work? 17,000 people showed up. 795 stayed to the end, which is close to what we’d expect from previous MOOC data but still a bit depressing. However, only 4,000 only tried to do the assignments and, in the warm up, a lot of people dropped out after the warm-up assignment. Looking at this, 1,884 completed the warm-up and stayed (got qualified), which makes the stay rate about 42%. (Hmm, not sure I agree with this numerical handling but I don’t have a better solution.)

Did students use the open framework for structure? It looks like there was revision behaviour, using the freedom of the openness to improve previous solutions with new knowledge. The actual participation rate was interesting because some students completed in 20, and some in 60.

Was it a success or a failure? Well, the students love it (yeah, you know how I feel about that kind of thing). Well, they surveyed the students at the end and they had realised that optimisation takes time (which is very, very true). The overall experience was positive despite the amount of work involved, and the course was rated as being hard.  The students were asked what their favourite part of the course and this was presented as a word cloud. Programming dominated (?!) followed by assignments (?!?!?!?!).

Their assignment choice was interesting because they deliberately chose examples that would work for one solution approach but not another. (For example, the Travelling Salesman Problem was provided at a scale where the Dynamic Programming solution wouldn’t fit into memory.)

There’s still a lot of dependency on this notion that “leaderboards are motivating”. From looking at the word cloud, which is a very high level approach, the students enjoyed the assignments and were happy to do programming, in a safe, retry-friendly (and hence failure tolerant) environment. In my opinion, the reminder of the work they’ve done is potentially more likely to be the reason they liked leader boards rather than as a motivating factor. (Time to set up a really good research study!)

Anyway, the final real session was a corker and I greatly enjoyed it! On to lunch and FRIED CHICKEN.

SIGCSE Day 3, “What We Say, What They Do”, Saturday, 9-10:15am, (#SIGCSE2014)

Posted: March 9, 2014 | Author: | Filed under: Education | Tags: , , , , , , , , , , , , , , , , , | 3 Comments

The first paper was “Metaphors we teach by” presented by Ben Shapiro from Tufts. What are the type of metaphors that CS1 instructors use and what are the wrinkles in these metaphors. What do we mean by metaphors? Ben’s talking about conceptual metaphors, linguistic devices to allow us to understand one idea in terms o another idea that we already know. Example: love is a journey – twists and turns, no guaranteed good ending, The structure of a metaphor is that you have a thing we’re trying to explain (the target) in terms of something we already know (the source).  Conceptual metaphors are explanatory devices to assist us in understanding new things.

Metaphors are widely used in teaching in CS, pointers, stacks and loops – all metaphorical aspects of computer science, but that’s not the focus of this study. How do people teach with metaphor? The authors couldn’t find any studies on general metaphor use in CS and its implication on student learning. An example from a birds-of-a-feather session held at this conference, a variable is like a box. A box can hold many different things but it holds things. (This has been the subject of a specific study.) Ben also introduced the “Too much milk” metaphor. This metaphor is laid out as follows. Jane comes home from work, goes to get milk from the fridge but her roommate has already drunk it (bad roommate!). Jane goes out to get more milk. While she’s out, her roommate comes back with milk, then Jane comes back with milk. Now they have too much milk! This could be used to explain race conditions in CS. Another example is the use of bus lockers mapping to virtual memory.
Ben returned to boxes again? One of the problems is that boxes can hold many things but a variable can only hold one thing – which appears to be a confusing point for learners who knew how boxes work. Is this a common problem? Metaphors have some benefits but come with this kind of baggage? Metaphors are partial mappings – they don’t match every aspect of the target to the source. (If it was a complete mapping they’d be the same thing.)
The research questions that the group considered were:
  • What metaphors do CS1 instructors use for teaching?
  • What are the trying to explain?
  • What are the sources that they use?
Learners don’t know where the mappings start and stop – where do the metaphors break down for students? What mistakes do they make because of these misunderstandings? Why does this matter? We all have knowledge on how to explain but we don’t have good published collections of the kind of metaphors that we use to teach CS, which would be handy for new teachers. We could study these and work out which are more effective. What are the most enduring and universal metaphors?
The study was interview-based, interviewing Uni-level CS1 instructors, ended up with 10 people, with an average of 13 years of teaching.  The interview questions given to these instructors were (paraphrased):
  • Levels taught and number of years
  • Tell me about a metahpor
  • Target to source mapping
  • Common questions students have
  • Where the metaphor breaks down
  • How to handle the breakdown in teaching.
Ben then presented the results. (We had a brief discussion of similes versus metaphors but I’ll leave that to you.) An instructor discussed using the simile of a portkey from Harry Potter to explain return statements in functions, because students had trouble with return existing immediately. The group of 10 people provided 18 different CS Concepts (Targets) and 19 Metaphorical Explanations (Sources).
What’s the target for “Card Catalogs”? Memory addressing and pointers. The results were interesting – there’s a wide range of ways to explain things! (The paper contains a table of a number of targets and sources.)
Out of date cultural references were identified as a problem and you have to be aware of the students’ cultural context. (Card desk and phone booths are nowhere near as widely used as they used to be.) Where do students make inferences beyond the metaphor? None of the 10 participants could give a single example of this happening! (This is surprising – Ben called it weird.) Two hypotheses – our metaphors are special and don’t get overextended (very unlikely) OR CS1 instructors poorly understand student thinking (more likely).
The following experimental studies may shed some light on this:
  • Which metaphors work better?
  • Cognitive clinical internviews, exploring how students think with metaphors and where incorrect inferences are drawn.
There was also a brief explanation of PCK (teachers’ pedagogical content knowledge) but I don’t have enough knowledge to fully flesh this out. Ben, if you’re reading this, feel free to add a beautifully explanatory comment. 🙂
The next walk was “‘Explain in Plain English’ Questions Revisited: Data Structures Problems” presented by Sue Fitzgerald and Laurie. This session opened with a poll to find out what the participants wanted and we all wanted to find out how to get students to use plain English. An Explain in Plain English  (EiPE) question asks you to describe what a chunk of code does, but not in a line by line discussion. A student’s ability to explain what a chink of code does correlates with a student’s ability to write and read code. The study wanted to investigate if this was just a novice phenomenon or if this advanced during the years and expertise. This study looked at 120 undergraduates in a CS2 course in data structures and algorithms using C++, with much more difficult questions than in earlier studies: linked lists, recursive calls and so on.
The students were given two questions in an exam with some preamble to describe the underlying class structure with a short example and a diagram. The students then had to look at a piece of code and determine what would happen in order to answer in the question as a plain English response. (There’s always a problem where you throw to an interactive response system where the question isn’t repeated, perhaps we need two screens.)
The SOLO taxonomy was used to analyse the problems (more Neo-Piagetian goodness!). Four of the SOLO categories were used: relational (summarises the code), multistructural (line by line explanation of the code) , unistructural (only describes one portion rather than the whole idea), and pre structural (misses it completely, gibberish). I was interested to see the examples presented, with pointers and mutual function calling, because it quickly became apparent that the room I was in (which had a lot of CS people in it) were having to think relatively hard about the answer to the second example. One of the things about working memory is that it’s not very deep and none of us were quite ready to work in a session 🙂 but a lot of good discussion ensued. The students would have had ready access to the preamble code but I do wonder how much obfuscation is really required here. The speaker made a parenthetical comment that experts usually doodle but where was our pen and paper! (As someone else said, reinforcing the point that we didn’t come prepared to work, nobody told us we had to bring paper. 🙂 ) We then got to classify a student response that was quite “student-y”. (A question came up as to whether an answer can be relational if it’s wrong – the opinion appears to be that a concise, complete and incorrect answer could be considered relational. A point for later discussion.) The answer we saw was multistructural because it was a line-by-line answer – it wasn’t clear, concise and abstract. We then saw another response that was much more terse but far less accurate. THe group tossed up between unistructural and pre structural. (The group couldn’t see the original code or the question, so this uncertainty make sense. Again, a problem with trying to have an engaging on-line response system and a presentation on the same screen. The presenters did a great job of trying to make it work but it’s not ideal.)
What about correlations? For the first question asked, students who gave relational and multistructural answers generally passed, with a 58% grade. Those who answered at the uni or pre level generally failed with an average grade of 38%. In the second test question, the relational and multi group generally passed with a grade of 61.2%, the uni and pre group generally failed with an achieved grade of 42%.
So these correlations hold for no-novice programmers. A mix of explaining, writing and reading code is an effective way to develop good programming skills and EiPE questions give students good practice in the valuable skills of explaining code. Instructors can overestimate how well students understand presented code – asking them to explain it back is very useful for student self-assessment. The authors’ speculation is that explaining code to peers is probably part of the success of peer instruction and pair programming.
The final talk was “A Formative Study of Influences on Student Testing Behaviours” presented by Kevin Buffardi, from VT. In their introductory CS1 and CS2 courses they use Test-Driven Development (TDD) – code a little, test a little, for incremental development. It’s popular in industry, so students come out with relevant experience, but some previous studies have found improvement in student work when they closely adhered to TDD philosophy. BUT a lot of students didn’t follow it at all! So the authors were looking for ways to encourage students to follow this, especially when they were on their own and programming by themselves. Because it’s a process, you can tell what happened by looking at the final program but they use WebCAT and so can track the developmental stages of the program as students submit their work for partial grading. These snapshots provide clear views of what the students are doing over time. (I really have to look at what we could do with WebCAT. Our existing automarker is getting a bit creaky.) Students also received hints back when they submitted their work, general and instructor level.
The first time students achieved something with any type of testing, they would get a “Good Start” feedback and be entitled to a free hint. If you kept up with your testing, you would ‘buy’ more hints. If your test coverage was good, you got more hints. If your coverage was poor, you got general feedback. (Prior to this, WebCAT only gave 3 hints. Now there are no free hints but you can buy an unlimited number.) This is an adaptive feedback mechanism, to encourage testing with hints as incentives. The study compared reinforcement treatments:
  • Constant – even time a goal achieve, you got a hint (Consistently rewards target behaviour)
  • Delayed – Hints when earned, at most one hint per hour (less inceptive for hammering the system)
  • Random – 50% chance of hints when goal is met. (Should reduce dependency on extrinsic behaviours)
Should you show them the goal or not? This was an additional factor – the goals were either visual (concrete goal) or obscured (suggest improvement without specified target). These were a paired treatment.
What was the impact? There were no differences in the number of lines written, but the visual goal lead to students getting better test coverage than obscured goal. There didn’t appear to be a long term effect but there is an upcoming ITiCSE talk that will discuss this further. There were some changes from one submission to another but this wasn’t covered in detail.
The authors held formative group interviews where the students explained their development process and interaction with WebCAT. They said that they valued several types of evaluation, they paid attention to RED progress bars (visualisation and dash boarding – I’d argue that is is more about awareness than motivation), and noticed when they earned a hint but didn’t get it. The students drew their individual developmental process as a diagram and, while everyone had a unique approach, but there were two general approaches. Test last approach showed up: write a solution, submit a solution to WebCAT, take a break, do some testing, then submit to WebCAT again. Periodic testing approach was the other pattern seen, where they wrote solutions, WebCAT, write tests, submit to WebCAT, then revise solution and tests, and iterate.
Going forward, the automated evaluation became part of their development strategy. There were conflicting interests: the correctness reports from WebCAT were actually reducing the need to write their own tests because they were getting an indication of how well it was working. This is an important point for me, because from the examples I saw, I really couldn’t see what I would call test-driven development, especially for test last, so the framework is not encouraging the right behaviour. Kevin handled my question on this well, because it’s a complicated issue, and I’m really looking forward to seeing the ITiCSE paper follow-up! Behavioural change is difficult and, as Kevin rightly noted, it’s optimistic to think that we can achieve it in the short term.
Everyone wants to get students doing the right thing but it’s a very complicated issue. Much food for thought and a great session!

SIGSCE Day 2, “Focus on K-12: Informal Education, Curriculum and Robots”, Paper 1, 3:45-5:00, (#SIGCSE2014)

Posted: March 8, 2014 | Author: | Filed under: Education | Tags: , , , , , , , , , , , , , , , , , , | Leave a comment

The first paper is “They can’t find us: The Search for Informal CS Education” by Betsy DiSalvo, Cecili Reid, Parisa Khanipour Roshan, all from Georgia Tech. (Mark wrote this paper up recently.) There are lots of resources around, MOOCs, on-line systems tools, Khan academy and Code Academy and, of course the aggregators. If all of this is here, why aren’t we getting the equalisation effects we expect?

Well, the wealth and the resource-aware actually know how to search and access these, and are more aware of them, so the inequality persists. The Marketing strategies are also pointed at this group, rather than targeting those needing educational equity. The cultural values of the audiences vary. (People think Scratch is a toy, rather than a useful and pragmatic real-world tool.) There’s also access – access to technical resource, social support for doing this and knowledge of the search terms. We can address this issues by research mechanisms to address the ignored community.

Children’s access to informal learning is through their parents so how their parents search make a big difference. How do they search? The authors set up a booth to ask 16 parents in the group how they would do it. 3 were disqualified for literacy or disability reasons (which is another issue). Only one person found a site that was relevant to CS education. Building from that, what are the search terms that they are using for computer learning and why aren’t hey coming up with good results. The terms that parents use supported this but the authors also used Google insights to see what other people were using. The most popular terms for the topic, the environment and the audience. Note: if you search for kids in computer learning you get fewer results than if you search for children in computer learning. The three terms that came up as being best were:

  • kids computer camp
  • kids computer classes
  • kids computer learning

The authors reviewed across some cities to see if there was variation by location for these search terse. What was the quality of these? 191 out of 840 search results were unique and relevant, with an average of 4.5 per search.

(As a note, MAN, does Betsy talk and present quickly. Completely comprehensible and great but really hard to transcribe!)

Results included : Camp, after school program, camp/afterschool, higher education, online activities, online classes/learning, directory results (often worse than Google), news, videos or social networks (again the quality was lower). Computer camps dominated what you could find on these search results – but these are not an option for low-income parents at $500/week so that’s not a really useful resource for them. Some came up for after school and higher ed in the large and midsize cities, but very little in the smaller cities. Unsurprisingly, smaller cities and lower socio-economic groups are not going to be able to find what they need to find, hence the inequality continues. There are many fine tools but NONE of them showed up on the 800+ results.

Without a background in CS or IT, you don’t know that these things exist and hence you can’t find it for your kids. Thus, these open educational resources are less accessible to these people, because they are only accessible through a mechanism that needs extra knowledge. (As a note, the authors only looked at the first two pages because “no-one looks past that”. 🙂 ) Other searches for things like kids maths learning, kids animal learning or kids physics learning turned up 48 out of 80 results (average of 16 unique results per search term), where 31 results were online, 101 had classes at uni – a big difference.

(These studies were carried out before code.org. Running the search again for kids computer learning does turn up code.org. Hooray, there is progress! If the study was run again, how much better would it be?)

We need to take a top down approach to provide standards for keywords and search terms, partnering with formal education and community programs. The MOOCs should talk to the Educational programming community, both could talk to the tutorial community and then we can throw in the Aggregators as well. Distant islands that don’t talk are just making this problem worse.

The bottom-up approach is getting an understanding of LSEO parenting, building communities and finding out how people search and making sure that we can handle it. Wow! Great talk but I think my head is going to explode!

During question time, someone asked why people aren’t more creative with their searches. This is, sadly, missing the point that, sitting in this community, we are empowered and skilled in searching. The whole point is that people outside of our community aren’t guaranteed to be able to find a way too be creative. I guess the first step is the same as for good teaching, putting ourselves in the heads of someone who is a true novice and helping to bring them to a more educated state.

 

 

SIGCSE Day 2, “Software Engineering: Courses”, Thursday, 1:45-3:00pm, (#SIGCSE2014)

Posted: March 8, 2014 | Author: | Filed under: Education | Tags: , , , , , , , , , , , , , | Leave a comment

Regrettably, despite best efforts, I was a bit late getting back from the lunch and I missed the opening session, so my apologies to Andres Neyem, Jose Benedetto and Andres Chacon, the authors of the first paper. From the discussion I heard, their course sounds interesting so I have to read their paper!

The next paper was “Selecting Open Source Software Projects to Teach Software Engineering” presented by Robert McCartney from University of Connecticut. The overview is why would we do this, the characteristics of the students, the projects and the course, finding good protects, what we found, how well it worked and what the conclusions were.

In terms of motivation, most of their SE course is in project work. The current project approach emphasises generative aspects. However, in most of SE, the effort involves maintenance and evolution. (Industry SE’s regularly tweak and tune, rather than build from the bottom.) The authors wanted to change focus to software maintenance and evolution, have the students working on an existing system, understanding it, adding enhancements, implementing, testing and documenting their changes. But if you’re going to do this, where do you get code from?

There are a lot of open source projects, available on0line, in a variety of domains and languages and at different stages of development. There should* be a project that fits every group. (*should not necessarily valid in this Universe.) The students are not actually being embedded in the open source community, the team is forking  the code and not planning to reintegrate it. The students themselves are in 2nd and 3rd year, with courses in OO and DS in Java, some experience with UML diagrams and Eclipse.

For each team of students, they get to pick a project from a set, try to understand the code, propose enhancements, describe and document  all o their plans, build their enhancements and present the results back. This happens over about 14 weeks. The language is Java and the code size has to be challenging but not impossible (so about 10K lines). The build time had to fit into a day or two of reasonable effort (which seems a little low to me – NF). Ideally, it should be a team-based project, where multiple developed could work in parallel. An initial look at the open source repositories on these criteria revealed a lot of issues: not many Java programs around 10K but Sourceforge showed promise. Interestingly, there were very few multi-developer projects around 10K lines. Choosing candidate projects located about 1000 candidates, where 200 actually met the initial size criterion. Having selected some, they added more criteria: had to be cool, recent, well documented, modular and have capacity to be built (no missing jar files, which turned out to be a big problem). Final number of projects: 19, size range 5.2-11 k lines.

That’s not a great figure. The takeaway? If you’re going to try and find projects for students, it’s going to take a while and the final yield is about 2%. Woo. The class ended up picking 16 projects and were able to comprehend the code (with staff help). Most of the enhancements, interestingly, involved GUIs. (Thats not so great, in my opinion, I’d always prefer to see functional additions first and shiny second.)

In concluding, Robert said that it’s possible to find OSS projects but it’s a lot of work. A search capability for OSS repositories would be really nice. Oh – now he’s talking about something else. Here it comes!

Small projects are not built and set up to the same standard as larger projects. They are harder to build, less-structured and lower quality documentation, most likely because it’s one person building it and they don’t notice the omissions. Thes second observation is that running more projects is harder for the staff. The lab supervisor ends up getting hammered. The response in later offerings was to offer fewer but larger projects (better design and well documented) and the lab supervisor can get away with learning fewer projects. On the next offering, they increased the project size (40-100K lines), gave the students the build information that was required (it’s frustrating without being amazingly educational). Overall, even with the same projects, teams produced different enhancements but with a lot less stress on the lab instructor.

Rather unfortunately, I had to duck out so I didn’t see Claudia’s final talk! I’ll write it up as a separate post later. (Claudia, you should probably re-present it at home. 🙂 )

SIGCSE, day 2, Le déjeuner des internationaux (#SIGCSE2014)

Posted: March 8, 2014 | Author: | Filed under: Education | Tags: , , , , , , , , , | Leave a comment

We had a lunch for the international contingent at SIGCSE, organised by Annemieke Craig from Deakin and Catherine Lang from Latrobe (late of Swinburne). There are apparently about 80 internationals here and we had about 24 at the lunch. Australians were over-represented but there were a lot of familiar faces and that’s always nice in a group of 1300 people.

Lots of fun and just one more benefit of a good conference. The group toasted Claudia Szabos’ success with the Best Paper award, again. We’re still having a lot of fun with that.

← Older Entries
Newer Entries →