We Expect Commitment – That’s Why We Have to Commit As Well.

Posted: May 18, 2012 | Author: | Filed under: Education | Tags: , , , , , , , , , , , , , , , , , , , | Leave a comment

I’m currently in Cupertino, California, to talk about how my University (or, to be precise, a Faculty in my University), starting using iPads in First Year by giving them to all starting students. As a result, last night I found myself at a large table on highly committed and passionate people in Education, talking about innovative support mechanisms for students.

Pizza and beer – Fuelling educational discussions since forever. (I love the Internet: I didn’t take any pictures of my food but a quick web-search for BJ’s Pizza Cupertino quickly turned up some good stuff.)

I’ve highlighted committed and passionate because it shows why those people are even at this meeting in the first place – they’re here to talk about something very cool that has been done for students, or a solution that has fixed a persistent or troublesome problem. From my conversations so far, everyone has been fascinated by what everyone else is doing and, in a couple of cases, I was taking notes furiously because it’s all great stuff that I want to do when I get home.

We expect our students to be committed to our courses: showing up, listening, contributing, collaborating, doing the work and getting the knowledge. We all clearly understand that passion makes that easier. Some students may have a sufficiently good view of where they want to go, when they come in, that we can draw on their goal drive to keep them going. However, a lot don’t, and even those who do have that view often turn out to have a slightly warped view of what their goal reality actually is. So, anything we can do to keep a student’s momentum going, while they work out what their goals and passions actually are, and make a true commitment to our courses, is really important.

And that’s where our commitment and passion come into things. As you may know, I travel a lot and, honestly, that’s pretty draining. However, after being awake for 33 hours after a trans-Pacific flight, I was still awake, alert and excited, sitting around last night talking to anyone who would listen with the things that we’re doing which are probably worth sharing. Much more importantly, I was fired up and interested to talk to the people around me who talking about the work that had been put in to make things work for students, the grand visions, the problems that had been overcome and, importantly, they could easily show me what they’d been doing because, in most cases, these systems are highly accessible in a mobile environment. Passion and commitment in my colleagues keeps me going and helps me to pass it on to my students.

Students always know if you’re into what you’re doing. Honestly, they do. Accepting that is one of the first steps to becoming a good teacher because it does away with that obstructive hypocrisy layer that bad teachers tend to cling to. This has to be more than a single teacher outlook though. Modern electronic systems for student support, learning and teaching, require the majority of educators to be involved in your institution. If you say “This is something you should do, please use it” and very few other lecturers do – who do the students believe? Because if they believe you, then your colleagues look bad (whether they should or not, I leave to you). If they believe your colleagues then you are wasting your effort and you’re going to get really frustrated. What about if half the class does and half doesn’t?

We’re going through some major strategic reviews at the moment back home and it’s really important that, whatever our new strategy on electronic support for learning and teaching is, it has to be something that the majority of staff and students can commit to, with results and participation drive or reward their passions. (It’s a good thing we’ve got some time to develop this, because it’s a really big ask!)

The educational times are most definitely a-changin’. (Sorry, I’m in California.) We’ve all seen what happens when new initiatives are pushed through, rather than guided through or introduced with strong support. Some time ago, I ran across a hierarchy of commitment that uses terms that I like, so I’m going to draw from that now. The terms are condemnation, complaint, compliance and commitment.

If we jam stuff through (new systems, new procedures, compulsory participation in on-line activities) without the proper consultation and preparation, we risk high levels of condemnation or under-mining from people who feel threatened or disenfranchised. Even if it’s not this bad, we may end up with people who just complain about it – Why should I? Who’s going to tell me to? Some people are just going to go along with it but, let’s face it, compliance is not the most inspirational mental state. Why are you doing it? Because someone told me to and I just thought I should go along with it.

We want commitment! I know what I’m doing. I agree with what I’m doing. I have chosen to not just take part but to do so willingly and I’m implicitly going to try and improve what’s going on. We want in our students, we want it in our colleagues. To get that in our colleagues for some of the new education systems is going to take a lot of discussion, a lot of thinking, a lot of careful design and some really good implementation, including honest and open review of what works and what doesn’t. It’s also going to take an honest and open discussion of the kind of workload involved to (a) produce everything properly as a set-up cost and (b) the ongoing costs in terms of workload, physical resources and time for staff, organisations and students.

So, if we want commitment from our students, then we must have commitment from our staff, which means that we who are involved in system planning and design have to commit in turn. I’m committed enough to come to California for about 8 more hours than I’m spending on planes here and back again. That, however, means nothing unless I show real commitment and take good things back to my own community, spend time and effort in carefully crafting effective communication for my students and colleagues, and keep on chasing it up and putting the effort in until something good is achieved.

Hurdles and Hang-ups: Identifying Those Things That Trip Up a Student

Posted: May 13, 2012 | Author: | Filed under: Education | Tags: , , , , , , , , , , , | 2 Comments

Most of the courses I teach have a number of guidelines in place that allow a student to, with relative ease and fairly early on, identify if they are meeting the requirements of the course. Some of these are based around their running assessment percentages, where a student knows their mark and can use this to estimate how they’re travelling. We use a minimum performance requirement that says that a student must achieve at least 40% in every component (where a component is “the examination” or “the aggregate mark across the whole of their programming assignments”) and 50% overall in order to pass. If a student doesn’t meet this, but would otherwise pass, we can look at targeted remedial (replacement) assessment in order to address the concern.

Bazinga! (With apologies to the Austrian athlete involved, who suffered no more injury than some cuts to his lower lip and jaw.)

One of the things we use is often referred to as a hurdle assessment, an assessment item that is compulsory and must be passed in order to pass the course. One of the good things about hurdle assessments is that you can take something that you consider to be a crucial skill and require a demonstration of adequate performance in that skill – well before the final examination and, often, in a way that is more practically oriented. Because of this we have practical programming exams early on in our course, to resolve the issues of students who can write about programming but can’t actually program yet.

It would be easy to think of these as barriers to progress, but the term hurdle is far more apt in this case, because if you visualise athletic training for the hurdles, you will see a sequence of hurdles leading to a goal. If you fall at one, then you require more training and then can attempt it again. This is another strong component of our guidelines – if we present hurdles, we must offer opportunities for learning and then reassessment.

Of course, this is the goal, role and burden of the educator: not the cheering on of the naturally gifted, but the encouragement, development and picking up of those who fall occasionally.

Picked correctly, hurdles identify a lack of ability or development in a core skill that is an absolute pre-requisite for further achievement. Picked poorly, it encourages misdirected effort, rote learning or eye-rolling by students as they undertake compulsory make-work.

I spend a lot to time trying to frame what is happening in the course so that my students can keep an eye on their own progress. A lot of what affects a student is nothing to do with academic ability and everything to do with their youth, their problems, their lives and their hang-ups. If I can provide some framing that tells them what is important and when it is important for it to be important, then I hope to provide a set of guides against which students can assess their own abilities and prioritise their efforts in order to achieve success.

Students have enough problems these days, with so many of them working or studying part-time or changing degrees or … well… 21st Century, really, without me adding to it by making the course a black box where no feedback or indicators reach them until I stamp a big red F on their paperwork and tell them to come back next year. If they can still afford it.

Whoops, I Seem To Have Written a Book. (A trip through Python and R Towards Truth)

Posted: May 6, 2012 | Author: | Filed under: Education | Tags: , , , , , , , , , , , | 3 Comments

Mark’s 1000th post (congratulations again!) and my own data analysis reminded me of something that I’ve been meaning to do for some time, which is work out how much I’ve written over the 151 published posts that I’ve managed this year. Now, foolish me, given that I can see the per-post word count, I started looking around to see how I could get an entire blog count.

And, while I’m sure it’s obvious to someone else who will immediately write in and say “Click here, Nick, sheesh!”, I couldn’t find anything that actually did what I wanted to do. So, being me, I decided to do it ye olde fashioned way – exporting the blog and analysing it manually. (Seriously, I know that it must be here somewhere but my brain decided that this would be a good time to try some analysis practice.)

Now, before I go on, here are the figures (not including this post!):

  • Since January 1st, I have published 151 posts. (Eek!)
  • The total number of words, including typed hyperlinks and image tags, is 102,136. (See previous eek.)
  • That’s an average of just over 676 words per post.

Is there a pattern to this? Have I increased the length of my posts over time as I gained confidence? Have they decreased over time as I got busier? Can I learn from this to make my posting more efficient?

The process was, unsurprisingly, not that simple because I took it as an opportunity to work on the design of an assignment for my Grand Challenges students. I deliberately started from scratch and assumed no installed software or programming knowledge above fundamentals on my part (this is harder than it sounds). Here are the steps:

  1. Double check for mechanisms to do this automatically.
  2. Realise that scraping 150 page counts by hand would be slow so I needed an alternative.
  3. Dump my WordPress site to an Export XML file.
  4. Stare at XML and slowly shake head. This would be hard to extract from without a good knowledge of Regular Expressions (which I was pretending not to have) or Python/Perl-fu (which I can pretend that I have to then not have but my Fu is weak these days).
  5. Drag Nathan Yau’s Visualize This down from the shelf of Design and Visualisation books in my study.
  6. Read Chapter 2, Handling Data.
  7. Download and install Beautiful Soup, an HTML and XML parsing package that does most of the hard word for you. (Instructions in Visualize This)
  8. Start Python
  9. Read the XML file into Python.
  10. Load up the Beautiful Soup package. (The version mentioned in the book is loaded up in a different way to mine so I had to re-enage my full programming brain to find the solution and make notes.)
  11. Mucked around until I extracted what I wanted to while using Python in interpreter mode (very, very cool and one of my favourite Python features).
  12. Wrote an 11 line program to do the extraction of the words, counting them and adding them (First year programming level, nothing fancy).

A number of you seasoned coders and educators out there will be staring at points 11 and 12, with a wavering finger, about to say “Hang on… have you just smoothed over about an hour plus of student activity?” Yes, I did. What took me a couple of minutes could easily be a 1-2 hour job for a student. Which is, of course, why it’s useful to do this because you find things like Beautiful Soup is called bs4 when it’s a locally installed module on OS X – which has obviously changed since Nathan wrote his book.

Now, a good play with data would be incomplete without a side trip into the tasty world of R. I dumped out the values that I obtained from word counting into a Comma Separated Value (CSV) file and, digging around in the R manual, Visualize This, and Data Analysis with Open Source Tools by Philipp Janert (O’Reilly), I did some really simple plotting. I wanted to see if there was any rhyme or reason to my posting, as a first cut. Here’s the first graph of words per post. The vertical axis is the number of words and the horizontal axis is the post number. So, reading left to right, you’ll see my development over time.

Words per Post

Sadly, there’s no pattern there at all – not only can’t we see one by eye, the correlation tests of R also give a big fat NO CORRELATION.

Now, here’s a graph of the moving average over a 5 day window, to see if there is another trend we can see. Maybe I do have trends, but they occur over a larger time?

Moving Average versus post

Uh, no. In fact, this one is worse for overall correlation. So there’s no real pattern here at all but there might be something lurking in the fine detail, because you can just about make out some peaks and troughs. (In fact, mucking around with the moving average window does show a pattern that I’ll talk about later.)

However, those of who you are used to reading graphs will have noticed something about the axis label for the x-axis. It’s labelled as wp$day. This would imply that I was plotting post day versus average or count and, of course, I’m not. There have not been 151 days since January the 1st, but there have been days when I have posted multiple times. At the moment, for a number of reasons, this isn’t clear to the reader. More importantly, the day on which I post is probably going to have a greater influence on me as I will have different access to the Internet and time available. During SIGCSE, I think I posted up to 6 times a day. Somewhere, this is lost in the structure of the data that considers each post as an independent entity. They consume time and, as a result, a longer post on the same day will reduce the chances of another long post on the same day – unless something unusual is going on.

There is a lot more analysis left to do here and it will take more time than I have today, unfortunately. But I’ll finish it off next week and get back to you, in case you’re interested.

What do I need to do next?

  1. Relabel my graphs so that it is much clearer what I am doing.
  2. If I am looking for structure, then I need to start looking at more obvious influences and, in this case, given there’s no other structure we can see, this probably means time-based grouping.
  3. I need to think what else I should include in determining a pattern to my posts. Weekday/weekend? Maybe my own calendar will tell me if I was travelling or really busy?
  4. Establish if there’s any reason for a pattern at all!

As a final note, novels ‘officially start at a count of 40,000 words, although they tend to fall into the 80-100,000 range. So, not only have I written a novel in the past 4 months, I am most likely on track to write two more by the end of the year, because I will produce roughly 160-180,000 more words this year. This is not the year of blogging, this is the year of a trilogy!

Next year, my blog posts will all be part of a rich saga involving a family of boy wizards who live on the wrong side on an Ice Wall next to a land that you just don’t walk into. On Mars. Look for it on Amazon. Thanks for reading!

Grand Challenges – A New Course and a New Program

Posted: May 4, 2012 | Author: | Filed under: Education | Tags: , , , , , , , , , , , , | Leave a comment

Oh, the poor students that I spoke to today. We have a new degree program starting, the Bachelor of Computer Science (Advanced), and it’s been given to me to coordinate and set up the first course: Grand Challenges in Computer Science, a first-year offering. This program (and all of its unique components) are aimed at students who have already demonstrated that they have got their academics sorted – a current GPA of 6 or higher (out of 7, that’s A equivalent or Distinctions for those who speak Australian), or an ATAR (Australian Tertiary Admission Rank) of 95+ out of 100. We identified some students who met the criteria and might want to be in the degree, and also sent out a general advertisement as some people were close and might make the criteria with a nudge.

These students know how to do their work and pass their courses. Because of this, we can assume some things and then build to a more advanced level.

Now, Nick, you might be saying, we all know that you’re (not so secretly) all about equality and accessibility. Why are you running this course that seems so… stratified?

Ah, well. Remember when I said you should probably feel sorry for them? I talked to these students about the current NSF Grand Challenges in CS, as I’ve already discussed, and pointed out that, given that the students in question had already displayed a degree of academic mastery, they could go further. In fact, they should be looking to go further. I told them that the course would be hard and that I would expect them to go further, challenge themselves and, as a reward, they’d do amazing things that they could add to their portfolios and their experience bucket.

I showed them that Cholera map and told them how smart data use saved lives. I showed them We Feel Fine and, after a slightly dud demo where everyone I clicked on had drug issues, I got them thinking about the sheer volume of data that is out there, waiting to be analysed, waiting to tell us important stories that will change the world. I pretty much asked them what they wanted to be, given that they’d already shown us what they were capable of. Did they want to go further?

There are so many things that we need, so many problems to solve, so much work to do. If I can get some good students interested in these problems early and provide a coursework system to help them to develop their solutions, then I can help them to make a difference. Do they have to? No, course entry is optional. But it’s so tempting. Small classes with a project-based assessment focus based on data visualisation: analysis, summarisation and visualisation in both static and dynamic areas. Introduction to relevant philosophy, cognitive fallacies, useful front-line analytics, and display languages like R and Processing (and maybe Julia). A chance to present to their colleagues, work with research groups, do student outreach – a chance to be creative and productive.

I, of course, will take as much of the course as I can, having worked on it with these students, and feed parts of it into outreach into schools, send other parts in different levels of our other degrees. Next year, I’ll write a brand new grand challenges course and do it all again. So this course is part of forming a new community core, a group of creative and accomplished leaders, to an extent, but it is also about making this infectious knowledge, a striving point for someone who now knows that a good mark will get them into a fascinating program. But I want all of it to be useful elsewhere, because if it’s good here, then (with enough scaffolding) it will be good elsewhere. Yes, I may have to slow it down elsewhere but that means that the work done here can help many courses in many ways.

I hope to get a good core of students and I’m really looking forward to seeing what they do. Are they up for the challenge? I guess we’ll find out at the end of second semester.

But, so you know, I think that they might be. Am I up for it?

I certainly hope so! 🙂

More on our image – and the need for a revamp.

Posted: April 28, 2012 | Author: | Filed under: Education | Tags: , , , , , , , | 4 Comments

Today, in between meetings with people about forming a cohesive ICT community and defining our identity, I saw a billboard as I walked along the streets of the Melbourne CBD.

A picture of a woman’s torso, naked except for a bra, with the slogan “Who said engineering was boring?”

Says it all, really, doesn’t it? I’ve long said that associating a verb in a sentence with a negative is the best way to get people to think about the verb rather than the more complex semantics of the negated verb. Now, for a whole lot of people, a vaguely leery billboard is going to put the words “engineering” and “boring” together.

Some of these people will be young people in our target recruitment group – mid to late school – and this kind of stuff sticks.

The building the billboard was on was built by civil engineers, using systems designed by mechanical and electronic/electrical engineers, the pictures were produced on machines constructed by computer systems engineers and elecs, images constructed and edited through digital cameras by tech-savvy photographers and processed on systems built by software engineers, computer scientists, electronic artists and many, many other people who are all being insulted by the same poster they helped to support and create. (My apologies because I didn’t list everybody, but the sheer scale of the number of people who contributed to that is quite large!)

Today, on my way home, a giant hunk of steel, powered by two big balls of spinning flame, climbed up into the sky and, in an hour, crossed a distance that used to take weeks to traverse. Right now, I am communicating with you around the world using a machine built of metal, burnt oil residue  and sand, that is sending information to you at nearly the speed of light, wherever you happen to be.

How, in anyone’s perverted lexicon, can that be anything other than exciting?

Got Vygotsky?

Posted: April 25, 2012 | Author: | Filed under: Education | Tags: , , , , , , , , , , , , , , , | 4 Comments

One of my colleagues drew my attention to an article in a recent Communications of the ACM, May 2012, vol 55, no 5, (Education: “Programming Goes to School” by Alexander Repenning) discussing how we can broaden participation of women and minorities in CS by integrating game design into middle school curricula (Thanks, Jocelyn!). The article itself is really interesting because it draws on a number of important theories in education and CS education but puts it together with a strong practical framework.

There’s a great diagram in it that shows Challenge versus Skills, and clearly illustrates that if you don’t get the challenge high enough, you get boredom. Set it too high, you get anxiety. In between the two, you have Flow (from Csíkszentmihályi’ s definition, where this indicates being fully immersed, feeling involved and successful) and the zone of proximal development (ZPD).

Extract of Figure from A. Repenning, "Programming Goes to School", CACM, 55, 5, May, 2012.

Which brings me to Vygotsky. Vgotsky’s conceptualisation of the zone of proximal development is designed to capture that continuum between the things that a learner can do with help, and the things that a learner can do without help. Looking at the diagram above, we can now see how learners can move from bored (when their skills exceed their challenges) into the Flow zone (where everything is in balance) but are can easily move into a space where they will need some help.

Most importantly, if we move upwards and out of the ZPD by increasing the challenge too soon, we reach the point where students start to realise that they are well beyond their comfort zone. What I like about the diagram above is that transition arrow from A to B that indicates the increase of skill and challenge that naturally traverses the ZPD but under control and in the expectation that we will return to the Flow zone again. Look at the red arrows – if we wait too long to give challenge on top of a dry skills base, the students get bored. It’s a nice way of putting together the knowledge that most of us already have – let’s do cool things sooner!

That’s one of the key aspects of educational activities – not they are all described in terms educational psychology but they show clear evidence of good design, with the clear vision of keeping students in an acceptably tolerable zone, even as we ramp up the challenges.

One the key quotes from the paper is:

The ability to create a playable game is essential if students are to reach a profound, personally changing “Wow, I can do this” realization.

If we’re looking to make our students think “I can do this”, then it’s essential to avoid the zone of anxiety where their positivity collapses under the weight of “I have no idea how anyone can even begin to help me to do this.” I like this short article and I really like the diagram – because it makes it very clear when we overburden with challenge, rather than building up skill and challenge in a matched way.

This Is What You Want, This Is What You Get

Posted: April 22, 2012 | Author: | Filed under: Education | Tags: , , , , , , | Leave a comment
via Metrojolt.comA poster with some of the lyrics for the Rolling Stones song "You can't always get what you want"

Midnight Mojo

One of the discussions that we seem to be having a lot is how the University will change in response to what students want, as we gain more flexibility in delivery and move away from face-to-face (bricks and mortar) to more blended approaches, possibly over distance learning. I’ve blogged a lot about this recently as I think about it but it’s always a lot more interesting to see what my colleagues think about it.

Some of my colleagues are, much like me, expecting things to be relatively similar after everything settles down. Books didn’t destroy academia, libraries didn’t remove the need for the lecturer, the tape recorder only goes so far. Yes, things may change, but we expect something familiar to remain. We’ll be able to reach more people because our learning offerings will accommodate more people.

Then there are people who seem to think that meeting student desire immediately means throwing all standards out the window. Somehow, there’s no halfway point between ‘no choice’ and ‘please take a degree as you leave’.

Of course, I’m presenting a straw man to discuss a straw man, but it’s a straw man that looks a lot like some that I’ve seen on campus. People who are designing their courses and systems to deal with the 0.1% of trouble makers rather than the vast majority of willing and able students.

There’s a point at which student desire can’t override our requirement for academic rigour and integrity. Frankly, there are many institutions out there that will sell you a degree but, of course, few people buy them expecting anything from them because everyone knows what kind of institutions they are. It boggles the mind that the few bad apples who show up at an accredited and ethical academy think that, somehow, only they will get the special treatment that they want and institutional quality will persist.

I have to work out what my students need from me and my University – based on what we told them we could do, what we can actually do (which is usually more than that) and what the student has the potential to do (which is usually more than they think they can do, once we’ve made them think about things a bit). There are many things that a student might want us to do, and we’ll have more flexibility for doing that in the future, but what they want isn’t always what they get. Sometimes, you get what you need.

(If you don’t have the Stones in your head right now, it’s time to go and buy some records.)

Why Teach Grand Challenges in ICT?

Posted: April 17, 2012 | Author: | Filed under: Education | Tags: , , , , , , | 2 Comments

I don’t normally drop in huge bits of texts from previous posts, but there are the NSF Committee for Cyberinfrastructure Task Force Grand Challenges, and their translation.

  1. Advanced Computational Methods and Algorithms
  2. High Performance Computing
  3. Software Infrastructure
  4. Data and Visualisation
  5. Education, Training and Workforce Development
  6. Grand Challenge Communities.

The same list in simpler, discipline free, terms:

  1. Better methods for solving hard problems.
  2. Big machines for solving hard problems.
  3. Good systems to run on the big machines, to support the better methods.
  4. Ways to see what results we have – people can see the results to make better decisions.
  5. Training people to make steps 1-4 work.
  6. Bring people together to make 1-5 work better with greater efficiency.

Why teach our students about these? Because they form the goals that we, as a discipline, will be striving for over the next few decades. Most of the items on this list are really, really hard to achieve. In explaining what we do, why we’re doing it, in tying our teaching into our professional practices and in giving authenticity to our entire educational approach – we need something large to aim at.

As an educator, knowing about the grand challenges in your own discipline shapes your ‘essential’ reading, gives you a hook to hang your lessons on and gives you, if we’re really waxing lyrical, a star to steer by. We all have something like this in our respective fields and it helps to show the overall direction and intention of our field.

This context shapes the things that you teach, the way that you teach and helps to ground students inside the professional aspects of what you’re talking about. It also helps you address those “Yes, but what use is this?” questions that beset us all.

It also sets our eyes up and out towards the horizon, to where the clouds, the sea, the sun and the sky fuse together and give us fantastic visions of what could be.

I Ran Out Of Time! (Why Are Software Estimates So Bad?)

Posted: March 24, 2012 | Author: | Filed under: Education | Tags: , , , , , , | Leave a comment

I read an interesting question on Quora regarding task estimation. The question, “Engineering Management: Why are software development task estimations regularly off by a factor of 2-3?“, had an answer wiki attached to it and then some quite long and specific answers. There is a lot there for anyone who works with students, and I’ll summarise some of them here that I like to talk about with my students:

  • The idea that if we plan hard enough, we can control everything: Planning gives us the illusion of control in many regards. If we are unrealistic in assessment of our own abilities or we don’t account for the unexpected, then we are almost bound for failure. Making big plans doesn’t work if we never come up with concrete milestones, allocate resources that we have and do something other than plan.
  • Poor discovery work: If you don’t look into what is actually required then you can’t achieve it. Doing any kind of task assessment without working out what you’re being asked to do, how long you have, what else you have to do and how you will know when you’re done is wasted effort.
  • Failure to assess previous projects: Learn from your successes and your failures! How much time did you allocate last time? Was it enough? No? ADD MORE TIME! How closely related are the two projects – if one is a subset of another what does this say for the time involved? Can you re-use elements from the previous project? Be critical of your previous work. What did you learn? What could you improve? What can you re-use? What do you need to never do again?
  • Big hands, little maps: There’s a great answer on the linked web page of drawing a broad line on Google maps at a high-level view and estimating the walking time for a trip. The devil is in the details! If you wave your hands in a broad way across a map it makes the task look simple. You need to get down to the appropriate level to make a good estimate – too far down, you get caught up in minutiae, too far up, you get a false impression of plain sailing.

I found it to be an interesting question with lots of informative answers and a delightful thought experiment of walking the California coast. I hope you like it too!

Graphs, DAGS and Inverted Pyramids: When Is a Prerequisite Not a Prerequisite?

Posted: March 22, 2012 | Author: | Filed under: Education | Tags: , , , , , , , | 1 Comment

I attended a very interesting talk at SIGCSE called “Bayesian Network Analysis of Computer Science Grade Distributions” (Anthony and Raney, Baldwin-Wallace College). Their fundamental question was how could they develop computational tools to increase the graduation rate of students in their 4 year degree. Motivated by a desire to make grade predictions, and catch students before they fall off, they started searching their records back to 1998 to find out if they could get some answers out of student performance data.

One of their questions was: Are the prerequisites actually prerequisite? If this is true, then there should be at least some sort of correlation between performance and attendance in a prerequisite course and the courses that depend upon it. I liked their approach because it took advantage of structures and data that they already had, and to which they applied a number of different analytical techniques.

They started from a graph of the prerequisites, which should be able to be built as something where you start from an entry subject and can progress all the way through to some sort of graduation point, but can only progress to later courses if you have the prereqs. (If we’re being Computer Science-y, prereq graphs can’t contain links that take you around in a loop and must be directed acyclic graphs (DAGs), but you can ignore that bit.) As it turns out, this structure can easily be converted to certain analytical structures, which makes the analysis a lot easier as we don’t have to justify any structural modification.

Using one approach, the researchers found that they could estimate a missing mark in the list of student marks to an accuracy of 77% – that is they correctly estimate the missing (A,B,C,D,F) grade 77% of the time, compared with 30% of the time if they don’t take the prereqs into account.

They presented a number of other interesting results but one that I found both informative and amusing was that they tried to use an automated network learning algorithm to pick the most influential course in assessing how a student will perform across their degree. However, as they said themselves, they didn’t constrain the order of their analysis – although CS400 might depend upon CS300 in the graph, their algorithm just saw them as connected. Because of this, the network learning picked their final year, top grade, course as the most likely indicator of good performance. Well, yes, if you get an A in CSC430 then you’ve probably done pretty well up until now. The machine learning involved didn’t have this requirement as a constraint so it just picked the best starting point – from its perspective. (I though that this really reinforced what the researchers were talking about – that finding the answer here was more than just correlation and throwing computing power at it. We had to really understand what we wanted to make sure we got the right answer.)

Machine learning is going to give you an answer, in many cases, but it’s always interesting to see how many implicit assumptions there are that we ignore. It’s like trying to build a pyramid by saying “Which stone is placed to indicate that we’ve finished”, realising it’s the capstone and putting that down on the ground and walking away. We, of course, have points requirements for degrees, so it gets worse because now you have to keep building and doing it upside down!

Picture of an upside down pyramid.

I’m certainly not criticising the researchers here – I love their work, I think that they’re very open about where they are trying to take this and I thought it was a really important point to drive home. Just because we see structures in a certain way, we always have to be careful how we explain them to machines because we need useful information that can be used in our real teaching worlds. The researchers are going to work on order-constrained network learning to refine this and I’m really looking forward to seeing the follow-up on this work.

I am also sketching out some similar analysis for my new PhD student to do when he starts in April. Oh, I hope he’s not reading this because he’s going to be very, VERY busy. 🙂

← Older Entries
Newer Entries →