I Ran Out Of Time! (Why Are Software Estimates So Bad?)

Posted: March 24, 2012 | Author: | Filed under: Education | Tags: , , , , , , | Leave a comment

I read an interesting question on Quora regarding task estimation. The question, “Engineering Management: Why are software development task estimations regularly off by a factor of 2-3?“, had an answer wiki attached to it and then some quite long and specific answers. There is a lot there for anyone who works with students, and I’ll summarise some of them here that I like to talk about with my students:

  • The idea that if we plan hard enough, we can control everything: Planning gives us the illusion of control in many regards. If we are unrealistic in assessment of our own abilities or we don’t account for the unexpected, then we are almost bound for failure. Making big plans doesn’t work if we never come up with concrete milestones, allocate resources that we have and do something other than plan.
  • Poor discovery work: If you don’t look into what is actually required then you can’t achieve it. Doing any kind of task assessment without working out what you’re being asked to do, how long you have, what else you have to do and how you will know when you’re done is wasted effort.
  • Failure to assess previous projects: Learn from your successes and your failures! How much time did you allocate last time? Was it enough? No? ADD MORE TIME! How closely related are the two projects – if one is a subset of another what does this say for the time involved? Can you re-use elements from the previous project? Be critical of your previous work. What did you learn? What could you improve? What can you re-use? What do you need to never do again?
  • Big hands, little maps: There’s a great answer on the linked web page of drawing a broad line on Google maps at a high-level view and estimating the walking time for a trip. The devil is in the details! If you wave your hands in a broad way across a map it makes the task look simple. You need to get down to the appropriate level to make a good estimate – too far down, you get caught up in minutiae, too far up, you get a false impression of plain sailing.

I found it to be an interesting question with lots of informative answers and a delightful thought experiment of walking the California coast. I hope you like it too!

How Do I Model The Students Who Leave?

Posted: March 23, 2012 | Author: | Filed under: Education | Tags: , , , , , , | Leave a comment

This is a quick note on one of the problems I face in trying to analyse student data: dealing with students who are only in the system so briefly that I can’t capture much data on them. In my other educational research work I can look at student behaviour in terms of final grades and on-time assignment submission but, in order to try and see the impact of what we’re doing on behaviour, I really have to be able to capture data before and after a change. I then have to try and eliminate all other factors to find a correlation that looks like it’s significant.

In yesterday’s post, I didn’t mention that one of the issues that the Baldwin-Wallace researchers noted was trying to deal with students who gave some initial data and then left the system – how do you incorporate these students in a way that allows you to infer behaviour without introducing the spectre of bias because you’ve inserted dummy data into your system. They had discussed adding another grade type, W or PW, that would allow them to keep students in their data who had left the program early – can you spot the situation that will lead to people leaving early and can we predict the withdrawal from the course based on earlier performance?

I face the same problem in a lot of my assignment submission data. I have 17,000 students in the initial dataset but, after cleaning and removing students who withdraw, that shrinks a lot. Regrettably, this also removes the students that I really want to work with – those who have withdrawn. We use a binary notation as an overview for on-time and late submission, so extending the sequence is straight-forward, but any time we extend the sequence we have to justify it very, very well to make sure that we haven’t introduced too much noise or bias.

There are a lot of good existing techniques and, of course, Bayesian analysis is once again our friend in many ways but I’m now looking at machine learning to provide a very simple two-component partitioning – can I learn to predict who will be in the incomplete group and who won’t? I have to do something about the ‘length’ of the submission history or the most obvious thing the machine will probably learn is that ‘short history == fail’. I’m looking forward to getting onto this research in the very near future, especially if it ca give me insight into those students who are only with us for a short time. I really need a tool and a model that will work within the first 2-3 weeks – it’s a challenge but a fun one.

Graphs, DAGS and Inverted Pyramids: When Is a Prerequisite Not a Prerequisite?

Posted: March 22, 2012 | Author: | Filed under: Education | Tags: , , , , , , , | 1 Comment

I attended a very interesting talk at SIGCSE called “Bayesian Network Analysis of Computer Science Grade Distributions” (Anthony and Raney, Baldwin-Wallace College). Their fundamental question was how could they develop computational tools to increase the graduation rate of students in their 4 year degree. Motivated by a desire to make grade predictions, and catch students before they fall off, they started searching their records back to 1998 to find out if they could get some answers out of student performance data.

One of their questions was: Are the prerequisites actually prerequisite? If this is true, then there should be at least some sort of correlation between performance and attendance in a prerequisite course and the courses that depend upon it. I liked their approach because it took advantage of structures and data that they already had, and to which they applied a number of different analytical techniques.

They started from a graph of the prerequisites, which should be able to be built as something where you start from an entry subject and can progress all the way through to some sort of graduation point, but can only progress to later courses if you have the prereqs. (If we’re being Computer Science-y, prereq graphs can’t contain links that take you around in a loop and must be directed acyclic graphs (DAGs), but you can ignore that bit.) As it turns out, this structure can easily be converted to certain analytical structures, which makes the analysis a lot easier as we don’t have to justify any structural modification.

Using one approach, the researchers found that they could estimate a missing mark in the list of student marks to an accuracy of 77% – that is they correctly estimate the missing (A,B,C,D,F) grade 77% of the time, compared with 30% of the time if they don’t take the prereqs into account.

They presented a number of other interesting results but one that I found both informative and amusing was that they tried to use an automated network learning algorithm to pick the most influential course in assessing how a student will perform across their degree. However, as they said themselves, they didn’t constrain the order of their analysis – although CS400 might depend upon CS300 in the graph, their algorithm just saw them as connected. Because of this, the network learning picked their final year, top grade, course as the most likely indicator of good performance. Well, yes, if you get an A in CSC430 then you’ve probably done pretty well up until now. The machine learning involved didn’t have this requirement as a constraint so it just picked the best starting point – from its perspective. (I though that this really reinforced what the researchers were talking about – that finding the answer here was more than just correlation and throwing computing power at it. We had to really understand what we wanted to make sure we got the right answer.)

Machine learning is going to give you an answer, in many cases, but it’s always interesting to see how many implicit assumptions there are that we ignore. It’s like trying to build a pyramid by saying “Which stone is placed to indicate that we’ve finished”, realising it’s the capstone and putting that down on the ground and walking away. We, of course, have points requirements for degrees, so it gets worse because now you have to keep building and doing it upside down!

Picture of an upside down pyramid.

I’m certainly not criticising the researchers here – I love their work, I think that they’re very open about where they are trying to take this and I thought it was a really important point to drive home. Just because we see structures in a certain way, we always have to be careful how we explain them to machines because we need useful information that can be used in our real teaching worlds. The researchers are going to work on order-constrained network learning to refine this and I’m really looking forward to seeing the follow-up on this work.

I am also sketching out some similar analysis for my new PhD student to do when he starts in April. Oh, I hope he’s not reading this because he’s going to be very, VERY busy. 🙂

What Do I Study? What Do I Do? Showing the Path

Posted: March 21, 2012 | Author: | Filed under: Education | Tags: , , , , | 3 Comments

One of the things I’ve learned from flying a lot is that it’s never as easy to get from one point to another as you think. There are regional hubs, legal connections, affiliations and the many intricacies of which routes are allowed into which countries. There’s a reason that you can either retain the services of a travel agent for a fee or spend a lot of your own time trying to work out the best way to get from A to B. It would be nice if you could fit everything onto one simple diagram and see the best way to go but, even without the commercial concerns, it’s a very hard problem to solve if you’re worried about efficiency rather than connectivity.

We allow our students a lot of latitude in picking their path through their degrees. Although we offer programs that have a core of prerequisites, there are many opportunities for electives – courses that they can pick and choose from. But, on many occasions, students look at the total number of points they require, and the year level, and pick based on interest or short-term goals, rather than any form of long-term vision.

Going back to our airline model, it’s like trying to get to to New York from Sydney by picking the cheapest flight that goes east. Thinking in one-step-ahead terms prevents you from realising the benefits of flights into longer-range hubs, special deals and the round-the-world flight. Technically, optimising your solution so that your next step is the ‘best’ from those available is a greedy algorithm – each step will be optimal but it’s not guaranteed to give you the best overall solution, just a solution.

What would be great is if we could present students with a simple flight path, a map, a poster or an interactive tool that allows them to see where they want to go, where they’re starting from, and how they could get there based on our courses. I’ve started sketching out some ideas based on this but complexity is proving to be a problem – as expected. I have some sketches of solutions and, when I have something that might be useful, I’ll share it here.

The ACID test: Teaching With Examples From Other Areas

Posted: March 20, 2012 | Author: | Filed under: Education | Tags: , , , , , , , | 2 Comments

I’ve just returned from teaching an intensive module on Distributed Systems – no, don’t go, I have a point for everyone, not just the Computer Scientists in the audience. Dealing with computations that take place over several computers can be tricky because it can be difficult for everyone to agree whether all the things that they wanted to happen have actually happened. Combine that with the problems that occur when two or more people try to change the same thing at the same time and we need a strong mechanism to deal with it. The properties that refer to this are usually represented with the acronym ACID.

A sign that says Danger Acid

We use something called a transactional model – what we’re trying to achieve either happens or, if there’s a problem, we make it as if it never occurred (we call this atomicity). When we make change we want to keep the overall system consistent with regard to some key requirements (consistency). If two things are happening at once, but could fail, we set it up so that they don’t take account of each other’s changes until we’re sure that they’ve finished and are going to hang around (isolation). Finally, speaking of hanging around, once we’ve made something stick, we want it to stay stuck – that’s durability.

Why have I covered this? Because I want you to understand how I can take ACID (atomicity, consistency, isolation, durability) out of computer science and make students think about it in a different framework – that of the legal system. Here’s the question I posed to my students as part of a tutorial:

“Using Transaction Properties (ACID), discuss whether a person simultaneously accused of two crimes should be tried, in both cases, as if only one crime had been committed.”

Now this doesn’t seem related, but the complex issues in the presumption of innocence, not declaring previous crimes until sentencing and the nature of appeal can lead to a quite complicated and involved discussion. I like to start students off by getting them to think about the problem individually and asking questions to clear up any definitional problems. Then they go to their neighbour, then into small groups of 4-5. By the time we’re done the rooms full of discussion and we bring it together to illustrate that thinking about the problem in this way gets us away from memorised jargon inside the originating discipline and forces students to describe the situation based on their understanding of the concept.

This is a third year course so the question is designed to make people think – there are some answers that are better than others but almost all pathways based on careful thought will head towards a good answer.

Stepping outside the original discipline can be fun and useful – just make sure that you’re keeping the analogies accurate, precise and not too far from the original material. Hope this is useful to you!

Fixing Misdirected Effort: Guiding, with the occasional shove.

Posted: March 19, 2012 | Author: | Filed under: Education | Tags: , , , , , , | Leave a comment

Concept image of the six most common questions and answers on a signpost.

My students have a lot of questions and my job is often as much about helping them find the right questions as it is about finding the answers. One of the most frustrating aspects of education is when people fixate on the wrong thing, or invest their effort into the wrong ventures. I talked before how I believed that far more students were procrastinators versus lazy, they invest their effort without thinking about the time that they need or all of the responsibilities that they have.

That’s why it gets frustrating when all of the effort that they can expend goes into the wrong pathway. I’ll give you a couple of examples. We run a forum where students receive e-mail notification and, because the forum is an official dissemination point, students are compulsorily enrolled into certain forums and will receive mail whenever discussion takes place. Every year, there’s at least one student who starts complaining about receiving ‘all this e-mail’. (Generally not more than 10 messages a week, except during busy times when it might rise to 20-30. Per week.) We then reply with the reasons we’re doing it. They argue. Of course, the e-mail load of the forum then does rise, because of the mailed complaints about the e-mail load – not to mention the investment of time. The student who complains that mail is wasting their time generally spends more time in that one exchange than processing the mail for the semester would have cost.

Another example is students trying to work out how much effort they can avoid in writing practical submissions. They’ll wait outside your office for an hour and talk for hours (if you let them) about whether they have to do this bit, or if they can take this shortcut, and what happens if I do that. Sitting down and trying it will take about 5 minutes but, because they’re fearful or haven’t fully understand how they can improve, they spend their time trying to dodge work and, anecdotally, it looks like some of these people invest more time in trying to avoid the work than actually doing the work would have required. This is, of course, ignoring the benefits of doing the work in terms of reinforcement and learning.

Then, of course, we have the curse of the Computer Science academic, that terror to the human eye – the plagiarised, patchwork monstrosity of Frankencode!

Frontispiece_to_Frankenstein_1831

Frankencoding, my term for the practice of trying to build software by Googling sections (I’m a classicist, or I’d call it Googlecoding) and jamming them together, is a major time waster here as well. If you design your software and built it up, you understand each piece and can debug it to get it working. If you surf news groups and chuck together bits and pieces that you don’t understand, your monster will rise up and lurch off to the village trailing disaster in its wake. Oh, and for the record, it’s really obvious to a marker when it has happened and even more obvious when we ask you WHY you did something – if you don’t know, you probably didn’t write it and “I got it off the Internet” attracts nothing good in the way of marks.

What I want to do is get effort focused on the right things. I know that my students regard most of their studies as a mild inconvenience, so I don’t want them spending what time they do devote to academia on the wrong things. This means that I have to try and direct discussions into useful pathways, handle the ‘what if I do this’ by saying ‘why don’t you go and try it. It’ll take 5 minutes’ (framed correctly) and by regularly checking for plagiarised code – including tests of understanding that accompany practical coding exercises such as test reports and design documents. Once again, understanding is paramount and wasted effort is not useful to anyone.

I like to think of it a gentle guidance in the right direction. With the occasional friendly, but firmer, shove when someone looks like they’re going seriously off the rails. After all, we have the same goals: none of us want to waste our time!

Participation: The Price of Success

Posted: March 18, 2012 | Author: | Filed under: Education | Tags: , , , , , , | 2 Comments

In my various roles I have to look at interesting areas like on-line learning and teaching delivery. One of the classic problems in this area is the success of the initiative to get educators and students alike to use the technology – at which point it melts because the level of participation rises so high that the finite underlying resources are exhausted. The resource was never designed as if everyone would want to, and then actually go on to, use it.

I have had someone, seriously, say to me that an on-line learning system would work much better if the students didn’t all try to use it at once.

The same problem, of course, occurs with educators. If no-one is participating in class then you’re pushing a giant rock up hill to make things happen and it’s more likely than not that a lot of what you’re saying and doing isn’t being taken in. If everyone is participating in class, you’ve jumped into the Ringmaster’s hat, you’re constantly fielding e-mails, forum messages and appointment requests. And, of course, at the end of the long day it’s easy to fall into that highly questionable, but periodically expressed, mode of thinking “universities would be great if there were fewer students around“.

A picture of bread and butter.

Students are our bread and butter. (Not literally!)

I’ve attached a picture of bread and butter to drive this point home. Students are what makes the University. Their participation, their enthusiasm, their attendance, their passion, their ennui, the good and the bad things they do. If the systems we build don’t work with our students, or the volume of students, or automatically excludes a group of students because we can any provide resources for 70% of them, then I think that we’ve got something wrong.

Having said that, I get ‘tight budgets’, I understand ‘district funding shortfall’ and I certainly sympathise with ‘very high workloads’. I’m not saying that people are giving up or doing the wrong thing in the face of all these factors, I’m talking about the understanding I’ve come to that the measure of my success as an educator is almost always linked to how much students want to talk to me about constructing knowledge, rather than than just doing assignment work.

It’s one of those things that, if I prepare for, makes my life easier and I can then view that work blip as a positive indicator, rather than go down the curmudgeonly professorial path of resenting the intrusion on my time. Let’s face it, attitude management is as (if not more) important for the lecturers in the class as it is for the students. You want to feel like you’re doing something useful, you’d like some positive feedback and you want to think that you’re making a difference. Framing increased participation as desirable and something that you plan for has certainly helped me manage the increased workload associated with it – because I take is a sign that my effort is paying off.

Post #100: Why I Haven’t Left My University

Posted: March 17, 2012 | Author: | Filed under: Education, Opinion | Tags: , , , , , | Leave a comment

In light of all of the posts from people telling us why they have left their jobs (Goldman Sachs, Google and the Empire, with the meme still rising), I wanted to spend my 100th post telling you why I’m not leaving my job.

  1. I’m not disillusioned. A lot of the “Why I Left” (WIL) posts talk about the authors discovering that their job wasn’t what it seemed, or that it had changed and the culture was gone, or that terrible things had happened and either evil Ring Lords had taken over their world or, in some cases, Evil Hobbits had killed the Benevolent Dictator. (Perspective is important.) Yes, University culture is changing but, firstly, not all change is bad and, secondly, a lot of positive change is taking place. Is this the job I thought it was when I started? Well, no, but that’s because I didn’t really understand what the job was. Education, knowledge, learning, teaching, research, integrity, persistence, excellence. Sometimes the framework it comes in can be irritating (matrix management I’m looking at you) but the core is solid and, because of that, the house stands. I’m now spending effort to get into positions where I can help that change occur in a good way and with a good goal.
  2. I don’t work for shareholders. Or, if I do, I work for 22 million of them.This is a big one. Most Universities in Australia are public Universities – government money, i.e. taxes, go to the universities to pay about half of their bills. Everyone who pays tax invests in the Universities that educates them and their children. Because we live in Australia, even if you can’t pay tax at the moment, then while it is not as equitable and accessible as it used to be (we could fix that, you know) it is still possible for people to go to college. Yes, it would be nice if it were free again but that certainly wouldn’t happen under a profit-driven shareholder vested model. I work for the people and, because of that, I have to be ready to educate anyone, anywhere, anytime. I don’t get to fail off a group of people because I’ve decided that they’re not smart enough for me – I need to look at what I need them to do and what they can do and get them from one place to the other. Maybe they need more help to get to that stage? That’s my job to work out as well, at my level. Some of them won’t make it, sure, but I never want it to be due to anything that I didn’t do.
  3. My job is fantastic.On a given day I can be discussing new developments in technology, encouraging a group of students to code, writing applications for my own research or getting time to stare at a wall and think about how to make the world a better place. Better yet, I have AMAZING ROBES OF POWER in which to do this in times of high celebration. Yes, every so often someone says “Those who can do, those who can’t teach” but I have been and I have done, and I continue to do, and now I also teach (I’ve posted in the past about authenticity). The most useful thing about that phrase is that, when it’s said seriously, you’ve just been saved a lot of effort in character assessment. 🙂
  4. I am a small part of a large community doing the most important job of allFrom kindergarten to PhD, the preparation and training of the next generation is one of the most important things that will ever get done. Since we developed writing, we’ve been able to scale our expert numbers up to match the number of trainees with increasing ability – first we had to copy by hand, then print and now we have electronic distribution. But we still need educators to complete the process of developing knowledge and enabling people to be able to receive and develop knowledge. But what we do is important because, without it, society goes away. Knowledge erodes. Things fall down. The machine stops.
  5. Every so often, someone says thank you. Every so often, one of my students comes back, covered in the dust of the real world and thanks me for what I’ve done. Yes, they often say things like “Wow, that thing you told me – did you know it was right?” but I know what they mean. All that sitting in lecture theatres and working on assignments – it had a purpose. That purpose was the right one. Thank you.

And that’s five good reasons why I’m still here.

Let’s Get Intense! (On the road again for intensive teaching)

Posted: March 16, 2012 | Author: | Filed under: Education | Tags: , , , , , , | Leave a comment

I’m sitting in Singapore as I write this, preparing for a weekend of intensive teaching. Our intensively delivered courses span months, as does a normal course, but features a combination of in-person and on-line instruction. The in-person segment is two weekends, over a month apart, where I go through half of the course with the students. We work 3 hours on Friday night, 6 hours on Saturday and 7 on Sunday. In between that time, we use electronic fora, assignments and, recently, tools like Piazza to keep the community together.

Intensive mode teaching presents some challenges, of course, but it does have its benefits. Yes, having students together for such an intense session does require you to structure your material well, but you can rely upon entire concepts being framed (primed) for students without the intervening week of ‘the rest of their lives’ abrading the principles. Then again, cognitive elastic being what it is, sometimes it needs to get stretched and relaxed in order to allow people to pick up a principle.

My guidelines for intensives are pretty straightforward and, as I prepare myself for teaching, I like to review them so I’ll share them with you here. The last time I was in Singapore, I talked about some specific teaching practices, these are more pleasantly wooly.

  1. Concepts shouldn’t span intensive weekend sessions. Pretty sensible here, try and keep the concepts in one day if possible. Yes, development over time is good, but we’re going to try and develop with minimal revision and then use priming and follow-up to address the other issues.
  2. Prime students for later work through questions, activities and assignments. Thinking time is not at a premium here, neither is digesting time, which brings us to…
  3. Follow-up on knowledge development with questions, activities and assignments. Teach it, ask for it, evaluate it, refine it, encourage knowledge.
  4. Break up your activities. Whatever the activity, don’t do it for more than a few sessions in a row. People get fatigued, then disengage, then get bored.
  5. Avoid passive lecturing. Wherever possible, involve the student. Get them to sketch their own answer, talk about them and tie it back to the core material. In the intensive I have already handed out all of the lecture notes – I have a record to refer to.
  6. Build on interesting questions to drive learning. Posit examples, get the class discussing (even gentle arguing) about the examples and force a need for them to get more knowledge.
  7. Shape their mental models through constant review of their in-class and on-line content. Correct, suggest, reinforce, reward. Operant conditioning is a powerful tool and it can be just as effective in an on-line setting.
  8. Follow-up in person if possible. Electronic communication can be cold and easy to misinterpret. If you can, set aside time to sit down to discuss problems that were raised between intensives to solve these ‘backwards and forwards’ e-mail or forum loops.
  9. Be active in whatever on-line spaces you provide. When you answer, people will ask more. The students should feel confident that you will answer and respond.
  10. Set aside some time each day to handle this class as if you had a physical lecture or consulting time with them every week. By making time to think about their work, their progression and their participation, you’ll make it easier to do all of the above.

I try to do all of this and I can be more or less successful but I know that the more of these I tick off, the better the experience for my students, so these as a set are always my goal.

Teaching in the data deluge

Posted: March 15, 2012 | Author: | Filed under: Education | Tags: , , , , | Leave a comment

I currently subscribe to a design magazine called Desktop and an article called Artifacts reminded me of the big difference between my college life and that of my students. The article discussed the collection of inspirational and reference documents, books and items kept by professional designers, with some illustrated examples, but highlighted the difference between the early days of modern design and now.

“There was a time, not long past, when seeking inspiration and influence was the challenge, not sorting it.”

The article goes on to note the difference between sharing a dog-eared copy of a design magazine, going to the library (which the authors define, tongue in cheek) and having to copy printed resources.

Of course, our students have gone through the same change. I had to go to the library,as a student, if I couldn’t buy the book. Locating was the problem. These days my students have to be able to classify, sift and order because there is so much information to hand. Of course, no all sources are equal and the implicit authority granted by physical publication now faces off against the ease of availability of the top ten hits on Google.

I find this more challenging in some ways but far more interesting in others. Yes, it’s easy to incorporate work that’s not your own but we are living in the data deluge – this fact allows us to have discussions about assessing quality, determining validity and authority and what plagiarism is and, importantly, how to avoid doing it, even unintentionally.

I much prefer a world where the problem is sifting. It forces us to look at far more interesting and important questions than “when does the library open” and allows us to spend our effort on understanding and using knowledge, without expending the majority of our effort on trying to locate already-published information. That whole “shoulders of giants” thing works a lot better when you can see the giants and find their shoulders! A surfeit of shoulders is a much better challenge to have.

← Older Entries
Newer Entries →