Whoops, I Seem To Have Written a Book. (A trip through Python and R Towards Truth)

Posted: May 6, 2012 | Author: | Filed under: Education | Tags: , , , , , , , , , , , | 3 Comments

Mark’s 1000th post (congratulations again!) and my own data analysis reminded me of something that I’ve been meaning to do for some time, which is work out how much I’ve written over the 151 published posts that I’ve managed this year. Now, foolish me, given that I can see the per-post word count, I started looking around to see how I could get an entire blog count.

And, while I’m sure it’s obvious to someone else who will immediately write in and say “Click here, Nick, sheesh!”, I couldn’t find anything that actually did what I wanted to do. So, being me, I decided to do it ye olde fashioned way – exporting the blog and analysing it manually. (Seriously, I know that it must be here somewhere but my brain decided that this would be a good time to try some analysis practice.)

Now, before I go on, here are the figures (not including this post!):

  • Since January 1st, I have published 151 posts. (Eek!)
  • The total number of words, including typed hyperlinks and image tags, is 102,136. (See previous eek.)
  • That’s an average of just over 676 words per post.

Is there a pattern to this? Have I increased the length of my posts over time as I gained confidence? Have they decreased over time as I got busier? Can I learn from this to make my posting more efficient?

The process was, unsurprisingly, not that simple because I took it as an opportunity to work on the design of an assignment for my Grand Challenges students. I deliberately started from scratch and assumed no installed software or programming knowledge above fundamentals on my part (this is harder than it sounds). Here are the steps:

  1. Double check for mechanisms to do this automatically.
  2. Realise that scraping 150 page counts by hand would be slow so I needed an alternative.
  3. Dump my WordPress site to an Export XML file.
  4. Stare at XML and slowly shake head. This would be hard to extract from without a good knowledge of Regular Expressions (which I was pretending not to have) or Python/Perl-fu (which I can pretend that I have to then not have but my Fu is weak these days).
  5. Drag Nathan Yau’s Visualize This down from the shelf of Design and Visualisation books in my study.
  6. Read Chapter 2, Handling Data.
  7. Download and install Beautiful Soup, an HTML and XML parsing package that does most of the hard word for you. (Instructions in Visualize This)
  8. Start Python
  9. Read the XML file into Python.
  10. Load up the Beautiful Soup package. (The version mentioned in the book is loaded up in a different way to mine so I had to re-enage my full programming brain to find the solution and make notes.)
  11. Mucked around until I extracted what I wanted to while using Python in interpreter mode (very, very cool and one of my favourite Python features).
  12. Wrote an 11 line program to do the extraction of the words, counting them and adding them (First year programming level, nothing fancy).

A number of you seasoned coders and educators out there will be staring at points 11 and 12, with a wavering finger, about to say “Hang on… have you just smoothed over about an hour plus of student activity?” Yes, I did. What took me a couple of minutes could easily be a 1-2 hour job for a student. Which is, of course, why it’s useful to do this because you find things like Beautiful Soup is called bs4 when it’s a locally installed module on OS X – which has obviously changed since Nathan wrote his book.

Now, a good play with data would be incomplete without a side trip into the tasty world of R. I dumped out the values that I obtained from word counting into a Comma Separated Value (CSV) file and, digging around in the R manual, Visualize This, and Data Analysis with Open Source Tools by Philipp Janert (O’Reilly), I did some really simple plotting. I wanted to see if there was any rhyme or reason to my posting, as a first cut. Here’s the first graph of words per post. The vertical axis is the number of words and the horizontal axis is the post number. So, reading left to right, you’ll see my development over time.

Words per Post

Sadly, there’s no pattern there at all – not only can’t we see one by eye, the correlation tests of R also give a big fat NO CORRELATION.

Now, here’s a graph of the moving average over a 5 day window, to see if there is another trend we can see. Maybe I do have trends, but they occur over a larger time?

Moving Average versus post

Uh, no. In fact, this one is worse for overall correlation. So there’s no real pattern here at all but there might be something lurking in the fine detail, because you can just about make out some peaks and troughs. (In fact, mucking around with the moving average window does show a pattern that I’ll talk about later.)

However, those of who you are used to reading graphs will have noticed something about the axis label for the x-axis. It’s labelled as wp$day. This would imply that I was plotting post day versus average or count and, of course, I’m not. There have not been 151 days since January the 1st, but there have been days when I have posted multiple times. At the moment, for a number of reasons, this isn’t clear to the reader. More importantly, the day on which I post is probably going to have a greater influence on me as I will have different access to the Internet and time available. During SIGCSE, I think I posted up to 6 times a day. Somewhere, this is lost in the structure of the data that considers each post as an independent entity. They consume time and, as a result, a longer post on the same day will reduce the chances of another long post on the same day – unless something unusual is going on.

There is a lot more analysis left to do here and it will take more time than I have today, unfortunately. But I’ll finish it off next week and get back to you, in case you’re interested.

What do I need to do next?

  1. Relabel my graphs so that it is much clearer what I am doing.
  2. If I am looking for structure, then I need to start looking at more obvious influences and, in this case, given there’s no other structure we can see, this probably means time-based grouping.
  3. I need to think what else I should include in determining a pattern to my posts. Weekday/weekend? Maybe my own calendar will tell me if I was travelling or really busy?
  4. Establish if there’s any reason for a pattern at all!

As a final note, novels ‘officially start at a count of 40,000 words, although they tend to fall into the 80-100,000 range. So, not only have I written a novel in the past 4 months, I am most likely on track to write two more by the end of the year, because I will produce roughly 160-180,000 more words this year. This is not the year of blogging, this is the year of a trilogy!

Next year, my blog posts will all be part of a rich saga involving a family of boy wizards who live on the wrong side on an Ice Wall next to a land that you just don’t walk into. On Mars. Look for it on Amazon. Thanks for reading!

Got Vygotsky?

Posted: April 25, 2012 | Author: | Filed under: Education | Tags: , , , , , , , , , , , , , , , | 4 Comments

One of my colleagues drew my attention to an article in a recent Communications of the ACM, May 2012, vol 55, no 5, (Education: “Programming Goes to School” by Alexander Repenning) discussing how we can broaden participation of women and minorities in CS by integrating game design into middle school curricula (Thanks, Jocelyn!). The article itself is really interesting because it draws on a number of important theories in education and CS education but puts it together with a strong practical framework.

There’s a great diagram in it that shows Challenge versus Skills, and clearly illustrates that if you don’t get the challenge high enough, you get boredom. Set it too high, you get anxiety. In between the two, you have Flow (from Csíkszentmihályi’ s definition, where this indicates being fully immersed, feeling involved and successful) and the zone of proximal development (ZPD).

Extract of Figure from A. Repenning, "Programming Goes to School", CACM, 55, 5, May, 2012.

Which brings me to Vygotsky. Vgotsky’s conceptualisation of the zone of proximal development is designed to capture that continuum between the things that a learner can do with help, and the things that a learner can do without help. Looking at the diagram above, we can now see how learners can move from bored (when their skills exceed their challenges) into the Flow zone (where everything is in balance) but are can easily move into a space where they will need some help.

Most importantly, if we move upwards and out of the ZPD by increasing the challenge too soon, we reach the point where students start to realise that they are well beyond their comfort zone. What I like about the diagram above is that transition arrow from A to B that indicates the increase of skill and challenge that naturally traverses the ZPD but under control and in the expectation that we will return to the Flow zone again. Look at the red arrows – if we wait too long to give challenge on top of a dry skills base, the students get bored. It’s a nice way of putting together the knowledge that most of us already have – let’s do cool things sooner!

That’s one of the key aspects of educational activities – not they are all described in terms educational psychology but they show clear evidence of good design, with the clear vision of keeping students in an acceptably tolerable zone, even as we ramp up the challenges.

One the key quotes from the paper is:

The ability to create a playable game is essential if students are to reach a profound, personally changing “Wow, I can do this” realization.

If we’re looking to make our students think “I can do this”, then it’s essential to avoid the zone of anxiety where their positivity collapses under the weight of “I have no idea how anyone can even begin to help me to do this.” I like this short article and I really like the diagram – because it makes it very clear when we overburden with challenge, rather than building up skill and challenge in a matched way.

Educational Software Systems: What are our requirements?

Posted: April 8, 2012 | Author: | Filed under: Education | Tags: , , , , , , , , , | 1 Comment

My recent evaluation of strategic IT issues in my faculty brought one thing very clearly to my attention. If we assume that the content that I (or a student or any other academic) creates should continue to be available to me, unless I assign my rights to someone else, then we have a problem if the storage mechanism used is closed and fee-based. If it’s closed (proprietary formats possibly with deliberate obfuscation or encryption, or remote storage with access controls) then I can’t easily get it out of the system unless the software provider lets me. If this whole arrangement (licensing or access) is based on a fee, the the worst possible situation is that, in order to access old materials/data/whatever, I have to continue to pay largish sums on money to keep using something that I created. So that got me thinking – what else do I expect (naively or not) of the systems that we use for education? I’m talking mostly about Learning Management Systems (LMS) here because that’s my current focus. Here are some cut down versions of my current thoughts.

  1. Modular: Not all schools, even in the same college, are the same. Some are big, some are small. Some need essays checked for plagiarism when submitted. Some just need a place to drop the assignments. Some need interactive quizzes! Some just need web pages. If we have a system that’s built out of modules then I can get the modules that I need and (if money is involved) match investment to requirement. Even in the open source community, there are issues of performance, management burden and complexity in the modules that you choose so customisation here is useful. Modularity also isolates the impact of faults. Well-designed modules are like water-tight doors on a ship – one module failing doesn’t sink the ship.
  2. Extensible: There are always going to be requirements particular to your school or college. If your system can be extended then you can adjust the system to meet your specific needs. If not, then you have to work around it. (You could also call this mutable but I prefer the improvement implicit in extensible). Now I’m not saying who should be doing these changes because that’s a whole other argument – the fact that it can be done is what’s important here.
  3. Open Storage: Whatever I create, I should be able to get to, export, import it back from other systems and hang onto – especially if we migrate to a different system and shut the old one down. I’m a great believer in keeping formats open and then, if your product is excellent, I’ll happily use it. If, at the start of our relationship, you say “Well, you give me your data and you’re going to have to pay us if you want to get it out, and pay us every year you want access to it.” then you are pretty much going to have to be the only game in town because I have no power or control in that relationship – and I’m the one who should be in control here. (Most people are getting really good about this now, which is good to see, but there are far too many examples of software where the same producer couldn’t maintain compatibility across two versions.)
  4. Efficient: Using these kinds of systems should save you time. Something I’ve created should be able to be re-used easily, live or die as quickly as I want and survive between upgrades. There should be a definite advantage to doing this – if not, why are we using this system?
  5. Robust: It should be strong. Students attack in waves and I have beautiful graphs to show that the day before an assignment is due it’s “STOP, Hammer time”, and weak or delicate systems will fail under this kind of load. Any system designer who has assumed an average and hardened the system up to 70% concurrent usage had better step back and add about 40% to that number to count for multiple accesses, dangling connections, staff use… Even more importantly, if I make a change to something (add an assignment, undertake an assignment, change a mark) it needs to stay CHANGED. Finally, I need to be able to undo things when and if they go wrong. Because things go wrong.
  6. Invisible: Ultimately, I shouldn’t notice the system, and nor should my students. I use it to create things but I focus on creation, not your system. My students use it to access things and perform actions but they should never notice the system itself. If you look out a window, you should never notice cracked panes, dirty glass, rotting wood or the accidental sandwiching of a bird between the double-glazed panes.

I realise that there is a lot of debate on Open versus Proprietary systems in the community and I have no wish to open that can of worms – for myself, I use a Mac (mostly on the FreeBSD side) and do my analysis work in R, not S, but then I use Illustrator, not the Gimp, for final touch-ups. I’m the poster boy for using stuff that works so I have no wish to force people to choose open or closed, or ‘whatever is in between’. But I’d be really interested to see what else belongs (or what doesn’t belong) on this list – what do you think?

Putting the pieces together: Constructive Feedback

Posted: April 6, 2012 | Author: | Filed under: Education | Tags: , , , , , , , | Leave a comment

Over the last couple of days, I’ve introduced a scenario that examines the commonly held belief that negative reinforcement can have more benefit than positive reinforcement. I’ve then shown you, via card tricks, why the way that you construct your experiment (or the way that you think about your world) means that this becomes a self-fulfilling prophecy.

Now let’s talk about making use of this. Remember this diagram?

This shows what happens if, when you occupy a defined good or bad state, you then redefine your world so that you lose all notion of degree and construct a false dichotomy through partitioning. We have to think about the fundamental question here – what do we actually want to achieve when we categorise activities as good or bad?

Realistically, most highly praiseworthy events are unusual – the A+. Similarly, the F should be less represented across our students than the P and the “I’ve done nothing, I don’t care, give me 0” should be as unlikely as the A+. But, instead of looking at outcomes (marks), let’s look at behaviours. Now I’m going to construct my feedback to students to address the behaviour or approach that they employed to achieve a certain outcome. This allows me to separate the B+ of a gifted student who didn’t really try from the B+ of a student who struggled and strived to achieve it. It also allows me to separate the F of someone who didn’t show up to class from the F of someone who didn’t understand the material because their study habits aren’t working. Finally, this allows me to separate the A+ of achievement from the A+ of plagiarism or cheating.

Under stress, I expect people to drop back down in their ability to reason and carry out higher-level thought. We could refer to Maslow or read any book on history. We could invoke Hick’s Law – that time taken to make decisions increases as function of the number of choices, and factor in pilot thresholds, decision making fatigue and all of that. Or we could remember what it was like to be undergraduates. 🙂

Let me break down student behaviours into unacceptable, acceptable and desirable behaviours, and allocate the same nomenclature that I did for cards. So the unacceptable are now the 2,3,4, the acceptable and 5-10, and the desirable are J-K. (The numbers don’t matter, I’m after the concept.) When I construct feedback for my students, I want to encourage certain behaviours and discourage others. Ideally, I want to even develop more of certain behaviours (putting more Kings in the deck, effectively). But what happens if I don’t discourage the unacceptable behaviours?

Under stress, under reduced cognitive function and reduced ability to make decisions, those higher level desirable behaviours are most likely going to suffer. Let us model the student under stress as a stacked deck, where the Q and K have been removed. Not only do we have less possibility for praise but we have an increased probability of the student making the wrong decision and using a 2,3,4 behaviour. It doesn’t matter that we’ve developed all of these great new behaviours (added Kings), if under stress they all get thrown out anyway.

So we also have to look at removing or reducing the likelihood of the unacceptable behaviours. A student who has no unacceptable behaviour left in their deck will, regardless of the pressure, always behave in an acceptable manner. If you can also add desirable behaviours then they have an increase chance of doing well. This is where targeted feedback comes in at the behavioural level. I don’t need to worry about acceptable behaviours and I should be very careful in my feedback to recognise that there is an asymmetric situation where any improvement from unacceptable is great but a decline from desirable to acceptable may be nowhere near as catastrophic. Moderation, thought and student knowledge are all very useful concepts here. But we can also involve the student in useful and scaleable ways.

Raising student awareness of how their own process works is important. We use reflective exercises midway through our end-of-first year programming course to develop student process awareness: what worked, what didn’t work, how can you improve. This gives students a framework for self-assessment that allows them to find their own 2s and their own Kings. By getting the students to do it, with feedback from us, they develop it in their own vocabulary and it makes it ideal for sharing with other students at a similar level. In fact, as the final exercise, we ask students to summarise their growth in their process awareness during the course and ask them for advice that they would give to other students about to start. Here are three examples:

Write code. Go beyond your practicals, explore the language, try things out for yourself. The more you code, the better you’ll get, so take every opportunity you can. 

If I could give advice to a future … student, I would tell them to test their code as often as possible, while writing it. Make sure that each stage of your code is flawless before going onto the next stage. 

If I could give one piece of advice to students …, it would be to start the coding early and to keep at it. Sometimes you can have a problem that you just can’t work out how to fix but if you start early you leave yourself time to be able to walk away and come back later. A lot of the time when you come back you will see the problem very quickly and wonder why you couldn’t see it before. 

There are a lot of Kings in the above advice – in fact, let’s promote them and call them Aces, because these are behaviours that, with practice, will become habits and they’ll give you a K at the same time as removing a 2. “Practise your coding skills” “Test your code thoroughly” “Start early” Students also framed their lessons in terms of what not to do. As one student reflected, he knew that it would take him four days to write the code, so he was wondering why he only started two days out – you could read the puzzlement in the words as he wrote them. He’d identified the problem and, by semester’s end, had removed that 2.

I’m not saying anything amazing or controversial here, but I hope that this has framed a common situation in a way that is useful to you. And you got to shout at cards, too. Bonus!

Card Shouting: Teaching Probability and Useful Feedback Techniques by… Yelling at Cards

Posted: April 5, 2012 | Author: | Filed under: Education | Tags: , , , , , , , , , | 2 Comments

I introduced a teaching tool I call ‘Card Shouting’ into some of my lectures a while ago – usually those dealing with Puzzle Based Learning or thinking techniques. What does Card Shouting do? It demonstrates probability, shows why experimental construction is important and engages your class for the rest of the lecture. So what is Card Shouting?

(As a note, I’ve been a bit rushed today and I may not have put this the best way that I could. I apologise for that in advance and look forward to constructive suggestions for clarity.)

Take a standard deck of cards, shuffle it and put it face down somewhere where the class can see it. (You either need very big cards or a document camera so that everyone can see this in a large class. In a small class a standard deck is fine.)

Then explain the rules, with a little patter. Here’s an example of what I say: “I’m sick and tired of losing at Poker! I’ve decided that I want to train a pack of cards to give me the cards I want. Which ones do I want? High cards, lots of them! What don’t I want? Low cards – I don’t want any really low cards.”

Pause to leaf through the deck and pull out a Ace, King and Queen. Also grab a 2,3 and 4. Any suit is fine. Hold up the Ace, King and Queen.

“Ok, these guys are my favourites. What I’m going to do, after this, is shuffle everything back in and draw a card from the top of the deck and flip it up so you can see it. If it’s one of these (A, K, Q), I want to hear positive things – I want ‘yeahs!’ and applause and good stuff. But if it’s one of these (hold up 2,3 and 4) I want boos and hisses and something negative. If it’s not either – well, we don’t do anything.” (Big tip: put a list of the accepted positive and negative phrases on the board. It’s much easier that having to lecture people on not swearing)

“Let’s find out what works best: praise or yelling- but we’re going to have measure our outcomes. I want to know if my training is working! If I draw a bad card and abuse it, I want to see improvement – I don’t want to see one of the bad cards on the next draw. If I draw a good card, and praise it, I want the deck to give me another good card – if it’s lower than a J, I’ve wasted my praise!” Then work through with the class how you can measure positive and negative reaction from the cards – we’ve already set up the basis for this in yesterday’s post, so here’s that diagram again.

In this case, we record a tick in the top left hand box if we get a good card and, after praising it, we get another good card. We put a tick in top right hand box if we draw a bad card, abuse the deck, and then get anything that is NOT a bad card. Similarly for the bottom row, if we praise it and the next card is not good, we put a mark in the bottom left hand box and, finally, if we abuse the card and the next card is still bad, we record a mark in the bottom right hand box.

Now you shuffle and start drawing cards, getting the reactions and then, based on the card after the reaction, record improvement or decline in the face of praise and abuse. If you draw something that’s neither good NOR bad to start with – you can ignore it. We only care about the situation when we have drawn a good or a bad card and what the immediate next card is.

The pattern should emerge relatively quickly. Your card shouting is working! If you get a card in the bad zone, you shout at it, and it’s 3 times as likely to improve. But wait, praise doesn’t do anything for these cards! You can praise a card and, again, roughly 3 times as often – it gets worse and drops out of the praise zone!

Not only have we trained the deck, we’ve shown that abuse is a far more powerful training tool!

Wait.

What? Why is this working? I discussed this in outline yesterday when I pointed out it’s a commonly held belief that negative reinforcement is more likely to cause beneficial outcome based on observation – but I didn’t say why.

Well, let’s think about the problem. First of all, to make the maths easy, I’m going to pull the Ace out of the deck. Sorry, Ace. Now I have 12 cards, 2 to K.

Let me redefine my good cards as J, Q, K. I’ve already defined my bad cards as 2, 3, 4. There are 12 different card values overall. Let me make the problem even simpler. If we encounter a good card, and praise it, what are the chances that I get another good card? (Let’s assume that we have put together a million shuffled decks, so the removal of one card doesn’t alter the probabilities much.)

Well, there are 3 good card values, out of 12, and 9 card values that aren’t good – so that’s 9/12 (75%) chance that I will get a not good card after a good card. What about if I only have 12 cards? Well, if I draw a K, I now only have the Q and the J left in my good card set, compared to 9 other cards. So I have a 9/11 chance of getting a worse card! 75% chance of the next card being worse is actually the best situation.

You can probably see where this is going. I have a similar chance of showing improvement after I’ve abused the deck for giving me a bad card.

Shouting at the cards had no effect on the situation at all – success and failure are defined so that the chances of them occurring serially (success following success, failure following failure) are unlikely but, more subtly, we extend the definition of bad when we have a good card – and vice versa! If we draw any card from 5 to 10 normally, we don’t care about it, but if we want to stay in the good zone, then these 6 cards get added to the bad cards temporarily – we have effectively redefined our standards and made it harder to get the good result. Here’s a diagram, with colour for emphasis, to show the usually neutral outcomes are grouped with the extreme.. No wonder we get the results we do! The rules that we use have determined what our outcome MUST look like.

So, that’s Card Shouting. It seems a bit silly but it allows you to start talking about measurement and illustrate that the way you define things is important. It introduces probability in terms of counting. (You can also ask lots of other questions like “What would happen if good was 8-K and bad was 2-7?”)

My final post on this, tomorrow, is going to tie this all together to talk about giving students feedback, and why we have to move beyond simple definitions of positive and negative, to focus on adding constructive behaviours and reducing destructive behaviours. We’re going to talk about adding some Kings and removing some deuces. See you then.

Grand Challenges in Education – When we say grand, we mean GRAND!

Posted: March 31, 2012 | Author: | Filed under: Education | Tags: , , , , , , , | 2 Comments

Highly coloured picture of a piano

Some time ago, Mark Guzdial posted on the Grand Challenges in the US National Educational Technology Plan. If I may summarise the four, huge, challenges, they were:

  1. A real-time, self-optimising difficulty-adjusting, interactive learning experience delivery system.
  2. A similarly high-end system for assessment of cross-discipline complex aspects of expertise and competencies.
  3. Integrated capture, aggregation, mining and sharing of content, learning and financial data across all platforms in near real-time.
  4. Identify the most effective principles of online learning systems and on/offline systems that produce equal or better results than conventional instruction in half the time and half the cost.

Wow. That’s one heck of a list. Compare that with the list of grand challenges from the March, 2011, report of National Science Foundation Advisory Committee for Cyberinfrastructure Task Force on Grand Challenges, which defines the grand challenge problems for my discipline, Computer (Cyber) Science and Engineering. By looking at some very complex problems, they arrived at the following list of areas in which great strides can, and should, be made:

  1. Advanced Computational Methods and Algorithms
  2. High Performance Computing
  3. Software Infrastructure
  4. Data and Visualisation
  5. Education, Training and Workforce Development
  6. Grand Challenge Communities.

Let me rewrite this last list in simpler, discipline free, terms:

  1. Better methods for solving hard problems.
  2. Big machines for solving hard problems.
  3. Good systems to run on the big machines, to support the better methods.
  4. Ways to see what results we have – people can see the results to make better decisions.
  5. Training people to make steps 1-4 work.
  6. Bring people together to make 1-5 work better with greater efficiency.

Now, lets look back at the four USNETP educational grand challenges to see if we can as easily form such a cohesive flow – we want to be able to see how it all works together.

  1. Smart learning systems.
  2. Smart assessment systems.
  3. Data and Visualisation. (Nick note: get into data and visualisation! 🙂 )
  4. Fusing the best of the old and the best of the new.

Now, the USNETP focus is on useful R&D and these challenges are part of their overall view of “they all combine to form the ultimate grand challenge problem in education: establishing an integrated, end-to-end real-time system for managing learning outcomes and costs across our entire education system at all levels. ” but what immediately leaps out at me are the steps 5 and 6 from the previous list. Rather than embed the training and community aspects somewhere in the rest of a document, why not embrace this at the same level if we’re talking about grand challenges in Education? That would give us:

  1. Training educators to make steps 1-4 work.
  2. Forming communities of practice to make 1-5 work better with greater efficiency.

Now these last two steps, of course, are what we’re doing with the conferences, the journals, the meetings and blogs like this but it makes a lot of sense when we see it inside my discipline, so it seems to make sense in the general field of education. There’s no doubt that these two last steps are easily as hard to manage at scale as the other projects, even interoperating with them. In fact, by making them huge challenges we increase their worth, justify effort and validate the research community built up around them. These are financially-sensitive times, where academics have to provide a value for their work. Allocating these important tasks to the grand challenge level recognises the difficulty, the uncertainty of being able to solve the problem and the sheer amount of work that may be involved.

These are, of course, only my thoughts and I have a great deal to learn in this space. I’m still searching for answers but if there’s a nice convenient report that says “Well, duh, Nick, we’re doing that right here, right now” I look forward to correction and enlightenment.

But, if it’s not already part of the USNETP grand challenges – what do you think? Should it be?

Teaching Tools (again): Balancing Price, Need and Accessibility.

Posted: March 30, 2012 | Author: | Filed under: Education | Tags: , , , , , , | 4 Comments

I’ve spoken before on open source and teaching tools but I’ve been reviewing some interesting data on textbook purchasing. As some of you may know, book purchases are dropping in many areas because students feel that they don’t need to (or can’t afford to) buy the text. Some of the price burden of textbooks is the size, printing and shipping costs associated, so eBooks, which can be and often are, cheaper should be addressing this problem.

Is that our experience? Well, we’re still collecting data but, anecdotally, no. Despite eBooks being substantially cheaper, students aren’t buying them in any greater numbers. (Early indications are that it may actually be less.)

Price is always going to be an issue. 60-70% of a large number is still a large number (to a student).

Need is an issue – do students need the book as a text or a reference? Will they be able to get by on lecture notes? How is the course structured? There are important equity issues associated with forcing a student to buy a book as you don’t know what they’ve had to give up to do that, and the resale market for secondhand books is not what it once was.

But one of the big concerns of my students is accessibility. They are well aware that buying an electronic book may give it to them in a very constrained form – a book that can only be read on one machine and may not survive upgrades, a book that may not have a useful search mechanism, a book where you can’t easily highlight the text. Worse, it may be a book that, sometime in the future, just stops working and can never be read again.

Yes, publishing companies are pouring millions of dollars into solving this problem but books are special in a very important way. Books enable knowledge transfer, they don’t own or restrict the knowledge transfer. When you produce a physical book, people can expend effort to do what they like. Make a house out of it, read it, re-index it, tear out the pages and put them together in print density order. None of this is possible with an eBook unless someone lets you. (Ok, you can build a house but it will use your laptop or tablet.)

I can’t help thinking that most of the effort seems to be going into providing the experience that publishing companies want us to have, in terms of usage, ownership and access – focusing on controlling us rather than enabling us. Perhaps this is the point we should address first?

(If you haven’t read my post on Hal Abelson’s talk, you might want to get to that after this.He talks a lot about the problems with the walled garden and his terminology,including the very useful term generative, is a very interesting read.)

Graphs, DAGS and Inverted Pyramids: When Is a Prerequisite Not a Prerequisite?

Posted: March 22, 2012 | Author: | Filed under: Education | Tags: , , , , , , , | 1 Comment

I attended a very interesting talk at SIGCSE called “Bayesian Network Analysis of Computer Science Grade Distributions” (Anthony and Raney, Baldwin-Wallace College). Their fundamental question was how could they develop computational tools to increase the graduation rate of students in their 4 year degree. Motivated by a desire to make grade predictions, and catch students before they fall off, they started searching their records back to 1998 to find out if they could get some answers out of student performance data.

One of their questions was: Are the prerequisites actually prerequisite? If this is true, then there should be at least some sort of correlation between performance and attendance in a prerequisite course and the courses that depend upon it. I liked their approach because it took advantage of structures and data that they already had, and to which they applied a number of different analytical techniques.

They started from a graph of the prerequisites, which should be able to be built as something where you start from an entry subject and can progress all the way through to some sort of graduation point, but can only progress to later courses if you have the prereqs. (If we’re being Computer Science-y, prereq graphs can’t contain links that take you around in a loop and must be directed acyclic graphs (DAGs), but you can ignore that bit.) As it turns out, this structure can easily be converted to certain analytical structures, which makes the analysis a lot easier as we don’t have to justify any structural modification.

Using one approach, the researchers found that they could estimate a missing mark in the list of student marks to an accuracy of 77% – that is they correctly estimate the missing (A,B,C,D,F) grade 77% of the time, compared with 30% of the time if they don’t take the prereqs into account.

They presented a number of other interesting results but one that I found both informative and amusing was that they tried to use an automated network learning algorithm to pick the most influential course in assessing how a student will perform across their degree. However, as they said themselves, they didn’t constrain the order of their analysis – although CS400 might depend upon CS300 in the graph, their algorithm just saw them as connected. Because of this, the network learning picked their final year, top grade, course as the most likely indicator of good performance. Well, yes, if you get an A in CSC430 then you’ve probably done pretty well up until now. The machine learning involved didn’t have this requirement as a constraint so it just picked the best starting point – from its perspective. (I though that this really reinforced what the researchers were talking about – that finding the answer here was more than just correlation and throwing computing power at it. We had to really understand what we wanted to make sure we got the right answer.)

Machine learning is going to give you an answer, in many cases, but it’s always interesting to see how many implicit assumptions there are that we ignore. It’s like trying to build a pyramid by saying “Which stone is placed to indicate that we’ve finished”, realising it’s the capstone and putting that down on the ground and walking away. We, of course, have points requirements for degrees, so it gets worse because now you have to keep building and doing it upside down!

Picture of an upside down pyramid.

I’m certainly not criticising the researchers here – I love their work, I think that they’re very open about where they are trying to take this and I thought it was a really important point to drive home. Just because we see structures in a certain way, we always have to be careful how we explain them to machines because we need useful information that can be used in our real teaching worlds. The researchers are going to work on order-constrained network learning to refine this and I’m really looking forward to seeing the follow-up on this work.

I am also sketching out some similar analysis for my new PhD student to do when he starts in April. Oh, I hope he’s not reading this because he’s going to be very, VERY busy. 🙂

The ACID test: Teaching With Examples From Other Areas

Posted: March 20, 2012 | Author: | Filed under: Education | Tags: , , , , , , , | 2 Comments

I’ve just returned from teaching an intensive module on Distributed Systems – no, don’t go, I have a point for everyone, not just the Computer Scientists in the audience. Dealing with computations that take place over several computers can be tricky because it can be difficult for everyone to agree whether all the things that they wanted to happen have actually happened. Combine that with the problems that occur when two or more people try to change the same thing at the same time and we need a strong mechanism to deal with it. The properties that refer to this are usually represented with the acronym ACID.

A sign that says Danger Acid

We use something called a transactional model – what we’re trying to achieve either happens or, if there’s a problem, we make it as if it never occurred (we call this atomicity). When we make change we want to keep the overall system consistent with regard to some key requirements (consistency). If two things are happening at once, but could fail, we set it up so that they don’t take account of each other’s changes until we’re sure that they’ve finished and are going to hang around (isolation). Finally, speaking of hanging around, once we’ve made something stick, we want it to stay stuck – that’s durability.

Why have I covered this? Because I want you to understand how I can take ACID (atomicity, consistency, isolation, durability) out of computer science and make students think about it in a different framework – that of the legal system. Here’s the question I posed to my students as part of a tutorial:

“Using Transaction Properties (ACID), discuss whether a person simultaneously accused of two crimes should be tried, in both cases, as if only one crime had been committed.”

Now this doesn’t seem related, but the complex issues in the presumption of innocence, not declaring previous crimes until sentencing and the nature of appeal can lead to a quite complicated and involved discussion. I like to start students off by getting them to think about the problem individually and asking questions to clear up any definitional problems. Then they go to their neighbour, then into small groups of 4-5. By the time we’re done the rooms full of discussion and we bring it together to illustrate that thinking about the problem in this way gets us away from memorised jargon inside the originating discipline and forces students to describe the situation based on their understanding of the concept.

This is a third year course so the question is designed to make people think – there are some answers that are better than others but almost all pathways based on careful thought will head towards a good answer.

Stepping outside the original discipline can be fun and useful – just make sure that you’re keeping the analogies accurate, precise and not too far from the original material. Hope this is useful to you!

Participation: The Price of Success

Posted: March 18, 2012 | Author: | Filed under: Education | Tags: , , , , , , | 2 Comments

In my various roles I have to look at interesting areas like on-line learning and teaching delivery. One of the classic problems in this area is the success of the initiative to get educators and students alike to use the technology – at which point it melts because the level of participation rises so high that the finite underlying resources are exhausted. The resource was never designed as if everyone would want to, and then actually go on to, use it.

I have had someone, seriously, say to me that an on-line learning system would work much better if the students didn’t all try to use it at once.

The same problem, of course, occurs with educators. If no-one is participating in class then you’re pushing a giant rock up hill to make things happen and it’s more likely than not that a lot of what you’re saying and doing isn’t being taken in. If everyone is participating in class, you’ve jumped into the Ringmaster’s hat, you’re constantly fielding e-mails, forum messages and appointment requests. And, of course, at the end of the long day it’s easy to fall into that highly questionable, but periodically expressed, mode of thinking “universities would be great if there were fewer students around“.

A picture of bread and butter.

Students are our bread and butter. (Not literally!)

I’ve attached a picture of bread and butter to drive this point home. Students are what makes the University. Their participation, their enthusiasm, their attendance, their passion, their ennui, the good and the bad things they do. If the systems we build don’t work with our students, or the volume of students, or automatically excludes a group of students because we can any provide resources for 70% of them, then I think that we’ve got something wrong.

Having said that, I get ‘tight budgets’, I understand ‘district funding shortfall’ and I certainly sympathise with ‘very high workloads’. I’m not saying that people are giving up or doing the wrong thing in the face of all these factors, I’m talking about the understanding I’ve come to that the measure of my success as an educator is almost always linked to how much students want to talk to me about constructing knowledge, rather than than just doing assignment work.

It’s one of those things that, if I prepare for, makes my life easier and I can then view that work blip as a positive indicator, rather than go down the curmudgeonly professorial path of resenting the intrusion on my time. Let’s face it, attitude management is as (if not more) important for the lecturers in the class as it is for the students. You want to feel like you’re doing something useful, you’d like some positive feedback and you want to think that you’re making a difference. Framing increased participation as desirable and something that you plan for has certainly helped me manage the increased workload associated with it – because I take is a sign that my effort is paying off.

← Older Entries
Newer Entries →