How Do I Model The Students Who Leave?
Posted: March 23, 2012 Filed under: Education | Tags: education, educational research, higher education, reflection, statistics, teaching, teaching approaches Leave a commentThis is a quick note on one of the problems I face in trying to analyse student data: dealing with students who are only in the system so briefly that I can’t capture much data on them. In my other educational research work I can look at student behaviour in terms of final grades and on-time assignment submission but, in order to try and see the impact of what we’re doing on behaviour, I really have to be able to capture data before and after a change. I then have to try and eliminate all other factors to find a correlation that looks like it’s significant.
In yesterday’s post, I didn’t mention that one of the issues that the Baldwin-Wallace researchers noted was trying to deal with students who gave some initial data and then left the system – how do you incorporate these students in a way that allows you to infer behaviour without introducing the spectre of bias because you’ve inserted dummy data into your system. They had discussed adding another grade type, W or PW, that would allow them to keep students in their data who had left the program early – can you spot the situation that will lead to people leaving early and can we predict the withdrawal from the course based on earlier performance?
I face the same problem in a lot of my assignment submission data. I have 17,000 students in the initial dataset but, after cleaning and removing students who withdraw, that shrinks a lot. Regrettably, this also removes the students that I really want to work with – those who have withdrawn. We use a binary notation as an overview for on-time and late submission, so extending the sequence is straight-forward, but any time we extend the sequence we have to justify it very, very well to make sure that we haven’t introduced too much noise or bias.
There are a lot of good existing techniques and, of course, Bayesian analysis is once again our friend in many ways but I’m now looking at machine learning to provide a very simple two-component partitioning – can I learn to predict who will be in the incomplete group and who won’t? I have to do something about the ‘length’ of the submission history or the most obvious thing the machine will probably learn is that ‘short history == fail’. I’m looking forward to getting onto this research in the very near future, especially if it ca give me insight into those students who are only with us for a short time. I really need a tool and a model that will work within the first 2-3 weeks – it’s a challenge but a fun one.
Graphs, DAGS and Inverted Pyramids: When Is a Prerequisite Not a Prerequisite?
Posted: March 22, 2012 Filed under: Education | Tags: curriculum, design, education, higher education, sigcse, teaching, teaching approaches, tools 1 CommentI attended a very interesting talk at SIGCSE called “Bayesian Network Analysis of Computer Science Grade Distributions” (Anthony and Raney, Baldwin-Wallace College). Their fundamental question was how could they develop computational tools to increase the graduation rate of students in their 4 year degree. Motivated by a desire to make grade predictions, and catch students before they fall off, they started searching their records back to 1998 to find out if they could get some answers out of student performance data.
One of their questions was: Are the prerequisites actually prerequisite? If this is true, then there should be at least some sort of correlation between performance and attendance in a prerequisite course and the courses that depend upon it. I liked their approach because it took advantage of structures and data that they already had, and to which they applied a number of different analytical techniques.
They started from a graph of the prerequisites, which should be able to be built as something where you start from an entry subject and can progress all the way through to some sort of graduation point, but can only progress to later courses if you have the prereqs. (If we’re being Computer Science-y, prereq graphs can’t contain links that take you around in a loop and must be directed acyclic graphs (DAGs), but you can ignore that bit.) As it turns out, this structure can easily be converted to certain analytical structures, which makes the analysis a lot easier as we don’t have to justify any structural modification.
Using one approach, the researchers found that they could estimate a missing mark in the list of student marks to an accuracy of 77% – that is they correctly estimate the missing (A,B,C,D,F) grade 77% of the time, compared with 30% of the time if they don’t take the prereqs into account.
They presented a number of other interesting results but one that I found both informative and amusing was that they tried to use an automated network learning algorithm to pick the most influential course in assessing how a student will perform across their degree. However, as they said themselves, they didn’t constrain the order of their analysis – although CS400 might depend upon CS300 in the graph, their algorithm just saw them as connected. Because of this, the network learning picked their final year, top grade, course as the most likely indicator of good performance. Well, yes, if you get an A in CSC430 then you’ve probably done pretty well up until now. The machine learning involved didn’t have this requirement as a constraint so it just picked the best starting point – from its perspective. (I though that this really reinforced what the researchers were talking about – that finding the answer here was more than just correlation and throwing computing power at it. We had to really understand what we wanted to make sure we got the right answer.)
Machine learning is going to give you an answer, in many cases, but it’s always interesting to see how many implicit assumptions there are that we ignore. It’s like trying to build a pyramid by saying “Which stone is placed to indicate that we’ve finished”, realising it’s the capstone and putting that down on the ground and walking away. We, of course, have points requirements for degrees, so it gets worse because now you have to keep building and doing it upside down!
I’m certainly not criticising the researchers here – I love their work, I think that they’re very open about where they are trying to take this and I thought it was a really important point to drive home. Just because we see structures in a certain way, we always have to be careful how we explain them to machines because we need useful information that can be used in our real teaching worlds. The researchers are going to work on order-constrained network learning to refine this and I’m really looking forward to seeing the follow-up on this work.
I am also sketching out some similar analysis for my new PhD student to do when he starts in April. Oh, I hope he’s not reading this because he’s going to be very, VERY busy. 🙂
What Do I Study? What Do I Do? Showing the Path
Posted: March 21, 2012 Filed under: Education | Tags: education, higher education, reflection, teaching, teaching approaches 3 CommentsOne of the things I’ve learned from flying a lot is that it’s never as easy to get from one point to another as you think. There are regional hubs, legal connections, affiliations and the many intricacies of which routes are allowed into which countries. There’s a reason that you can either retain the services of a travel agent for a fee or spend a lot of your own time trying to work out the best way to get from A to B. It would be nice if you could fit everything onto one simple diagram and see the best way to go but, even without the commercial concerns, it’s a very hard problem to solve if you’re worried about efficiency rather than connectivity.
We allow our students a lot of latitude in picking their path through their degrees. Although we offer programs that have a core of prerequisites, there are many opportunities for electives – courses that they can pick and choose from. But, on many occasions, students look at the total number of points they require, and the year level, and pick based on interest or short-term goals, rather than any form of long-term vision.
Going back to our airline model, it’s like trying to get to to New York from Sydney by picking the cheapest flight that goes east. Thinking in one-step-ahead terms prevents you from realising the benefits of flights into longer-range hubs, special deals and the round-the-world flight. Technically, optimising your solution so that your next step is the ‘best’ from those available is a greedy algorithm – each step will be optimal but it’s not guaranteed to give you the best overall solution, just a solution.
What would be great is if we could present students with a simple flight path, a map, a poster or an interactive tool that allows them to see where they want to go, where they’re starting from, and how they could get there based on our courses. I’ve started sketching out some ideas based on this but complexity is proving to be a problem – as expected. I have some sketches of solutions and, when I have something that might be useful, I’ll share it here.
The ACID test: Teaching With Examples From Other Areas
Posted: March 20, 2012 Filed under: Education | Tags: collaboration, education, higher education, principles of design, resources, teaching, teaching approaches, tools 2 CommentsI’ve just returned from teaching an intensive module on Distributed Systems – no, don’t go, I have a point for everyone, not just the Computer Scientists in the audience. Dealing with computations that take place over several computers can be tricky because it can be difficult for everyone to agree whether all the things that they wanted to happen have actually happened. Combine that with the problems that occur when two or more people try to change the same thing at the same time and we need a strong mechanism to deal with it. The properties that refer to this are usually represented with the acronym ACID.
We use something called a transactional model – what we’re trying to achieve either happens or, if there’s a problem, we make it as if it never occurred (we call this atomicity). When we make change we want to keep the overall system consistent with regard to some key requirements (consistency). If two things are happening at once, but could fail, we set it up so that they don’t take account of each other’s changes until we’re sure that they’ve finished and are going to hang around (isolation). Finally, speaking of hanging around, once we’ve made something stick, we want it to stay stuck – that’s durability.
Why have I covered this? Because I want you to understand how I can take ACID (atomicity, consistency, isolation, durability) out of computer science and make students think about it in a different framework – that of the legal system. Here’s the question I posed to my students as part of a tutorial:
“Using Transaction Properties (ACID), discuss whether a person simultaneously accused of two crimes should be tried, in both cases, as if only one crime had been committed.”
Now this doesn’t seem related, but the complex issues in the presumption of innocence, not declaring previous crimes until sentencing and the nature of appeal can lead to a quite complicated and involved discussion. I like to start students off by getting them to think about the problem individually and asking questions to clear up any definitional problems. Then they go to their neighbour, then into small groups of 4-5. By the time we’re done the rooms full of discussion and we bring it together to illustrate that thinking about the problem in this way gets us away from memorised jargon inside the originating discipline and forces students to describe the situation based on their understanding of the concept.
This is a third year course so the question is designed to make people think – there are some answers that are better than others but almost all pathways based on careful thought will head towards a good answer.
Stepping outside the original discipline can be fun and useful – just make sure that you’re keeping the analogies accurate, precise and not too far from the original material. Hope this is useful to you!
Fixing Misdirected Effort: Guiding, with the occasional shove.
Posted: March 19, 2012 Filed under: Education | Tags: education, educational problem, frankenstein, higher education, resources, teaching, teaching approaches Leave a comment
My students have a lot of questions and my job is often as much about helping them find the right questions as it is about finding the answers. One of the most frustrating aspects of education is when people fixate on the wrong thing, or invest their effort into the wrong ventures. I talked before how I believed that far more students were procrastinators versus lazy, they invest their effort without thinking about the time that they need or all of the responsibilities that they have.
That’s why it gets frustrating when all of the effort that they can expend goes into the wrong pathway. I’ll give you a couple of examples. We run a forum where students receive e-mail notification and, because the forum is an official dissemination point, students are compulsorily enrolled into certain forums and will receive mail whenever discussion takes place. Every year, there’s at least one student who starts complaining about receiving ‘all this e-mail’. (Generally not more than 10 messages a week, except during busy times when it might rise to 20-30. Per week.) We then reply with the reasons we’re doing it. They argue. Of course, the e-mail load of the forum then does rise, because of the mailed complaints about the e-mail load – not to mention the investment of time. The student who complains that mail is wasting their time generally spends more time in that one exchange than processing the mail for the semester would have cost.
Another example is students trying to work out how much effort they can avoid in writing practical submissions. They’ll wait outside your office for an hour and talk for hours (if you let them) about whether they have to do this bit, or if they can take this shortcut, and what happens if I do that. Sitting down and trying it will take about 5 minutes but, because they’re fearful or haven’t fully understand how they can improve, they spend their time trying to dodge work and, anecdotally, it looks like some of these people invest more time in trying to avoid the work than actually doing the work would have required. This is, of course, ignoring the benefits of doing the work in terms of reinforcement and learning.
Then, of course, we have the curse of the Computer Science academic, that terror to the human eye – the plagiarised, patchwork monstrosity of Frankencode!
Frankencoding, my term for the practice of trying to build software by Googling sections (I’m a classicist, or I’d call it Googlecoding) and jamming them together, is a major time waster here as well. If you design your software and built it up, you understand each piece and can debug it to get it working. If you surf news groups and chuck together bits and pieces that you don’t understand, your monster will rise up and lurch off to the village trailing disaster in its wake. Oh, and for the record, it’s really obvious to a marker when it has happened and even more obvious when we ask you WHY you did something – if you don’t know, you probably didn’t write it and “I got it off the Internet” attracts nothing good in the way of marks.
What I want to do is get effort focused on the right things. I know that my students regard most of their studies as a mild inconvenience, so I don’t want them spending what time they do devote to academia on the wrong things. This means that I have to try and direct discussions into useful pathways, handle the ‘what if I do this’ by saying ‘why don’t you go and try it. It’ll take 5 minutes’ (framed correctly) and by regularly checking for plagiarised code – including tests of understanding that accompany practical coding exercises such as test reports and design documents. Once again, understanding is paramount and wasted effort is not useful to anyone.
I like to think of it a gentle guidance in the right direction. With the occasional friendly, but firmer, shove when someone looks like they’re going seriously off the rails. After all, we have the same goals: none of us want to waste our time!
Participation: The Price of Success
Posted: March 18, 2012 Filed under: Education | Tags: education, higher education, reflection, teaching, teaching approaches, tools, workload 2 CommentsIn my various roles I have to look at interesting areas like on-line learning and teaching delivery. One of the classic problems in this area is the success of the initiative to get educators and students alike to use the technology – at which point it melts because the level of participation rises so high that the finite underlying resources are exhausted. The resource was never designed as if everyone would want to, and then actually go on to, use it.
I have had someone, seriously, say to me that an on-line learning system would work much better if the students didn’t all try to use it at once.
The same problem, of course, occurs with educators. If no-one is participating in class then you’re pushing a giant rock up hill to make things happen and it’s more likely than not that a lot of what you’re saying and doing isn’t being taken in. If everyone is participating in class, you’ve jumped into the Ringmaster’s hat, you’re constantly fielding e-mails, forum messages and appointment requests. And, of course, at the end of the long day it’s easy to fall into that highly questionable, but periodically expressed, mode of thinking “universities would be great if there were fewer students around“.
Students are our bread and butter. (Not literally!)
I’ve attached a picture of bread and butter to drive this point home. Students are what makes the University. Their participation, their enthusiasm, their attendance, their passion, their ennui, the good and the bad things they do. If the systems we build don’t work with our students, or the volume of students, or automatically excludes a group of students because we can any provide resources for 70% of them, then I think that we’ve got something wrong.
Having said that, I get ‘tight budgets’, I understand ‘district funding shortfall’ and I certainly sympathise with ‘very high workloads’. I’m not saying that people are giving up or doing the wrong thing in the face of all these factors, I’m talking about the understanding I’ve come to that the measure of my success as an educator is almost always linked to how much students want to talk to me about constructing knowledge, rather than than just doing assignment work.
It’s one of those things that, if I prepare for, makes my life easier and I can then view that work blip as a positive indicator, rather than go down the curmudgeonly professorial path of resenting the intrusion on my time. Let’s face it, attitude management is as (if not more) important for the lecturers in the class as it is for the students. You want to feel like you’re doing something useful, you’d like some positive feedback and you want to think that you’re making a difference. Framing increased participation as desirable and something that you plan for has certainly helped me manage the increased workload associated with it – because I take is a sign that my effort is paying off.
Let’s Get Intense! (On the road again for intensive teaching)
Posted: March 16, 2012 Filed under: Education | Tags: design, education, higher education, resources, teaching, teaching approaches, tools Leave a commentI’m sitting in Singapore as I write this, preparing for a weekend of intensive teaching. Our intensively delivered courses span months, as does a normal course, but features a combination of in-person and on-line instruction. The in-person segment is two weekends, over a month apart, where I go through half of the course with the students. We work 3 hours on Friday night, 6 hours on Saturday and 7 on Sunday. In between that time, we use electronic fora, assignments and, recently, tools like Piazza to keep the community together.
Intensive mode teaching presents some challenges, of course, but it does have its benefits. Yes, having students together for such an intense session does require you to structure your material well, but you can rely upon entire concepts being framed (primed) for students without the intervening week of ‘the rest of their lives’ abrading the principles. Then again, cognitive elastic being what it is, sometimes it needs to get stretched and relaxed in order to allow people to pick up a principle.
My guidelines for intensives are pretty straightforward and, as I prepare myself for teaching, I like to review them so I’ll share them with you here. The last time I was in Singapore, I talked about some specific teaching practices, these are more pleasantly wooly.
- Concepts shouldn’t span intensive weekend sessions. Pretty sensible here, try and keep the concepts in one day if possible. Yes, development over time is good, but we’re going to try and develop with minimal revision and then use priming and follow-up to address the other issues.
- Prime students for later work through questions, activities and assignments. Thinking time is not at a premium here, neither is digesting time, which brings us to…
- Follow-up on knowledge development with questions, activities and assignments. Teach it, ask for it, evaluate it, refine it, encourage knowledge.
- Break up your activities. Whatever the activity, don’t do it for more than a few sessions in a row. People get fatigued, then disengage, then get bored.
- Avoid passive lecturing. Wherever possible, involve the student. Get them to sketch their own answer, talk about them and tie it back to the core material. In the intensive I have already handed out all of the lecture notes – I have a record to refer to.
- Build on interesting questions to drive learning. Posit examples, get the class discussing (even gentle arguing) about the examples and force a need for them to get more knowledge.
- Shape their mental models through constant review of their in-class and on-line content. Correct, suggest, reinforce, reward. Operant conditioning is a powerful tool and it can be just as effective in an on-line setting.
- Follow-up in person if possible. Electronic communication can be cold and easy to misinterpret. If you can, set aside time to sit down to discuss problems that were raised between intensives to solve these ‘backwards and forwards’ e-mail or forum loops.
- Be active in whatever on-line spaces you provide. When you answer, people will ask more. The students should feel confident that you will answer and respond.
- Set aside some time each day to handle this class as if you had a physical lecture or consulting time with them every week. By making time to think about their work, their progression and their participation, you’ll make it easier to do all of the above.
I try to do all of this and I can be more or less successful but I know that the more of these I tick off, the better the experience for my students, so these as a set are always my goal.
Teaching in the data deluge
Posted: March 15, 2012 Filed under: Education | Tags: education, higher education, teaching, teaching approaches, tools Leave a commentI currently subscribe to a design magazine called Desktop and an article called Artifacts reminded me of the big difference between my college life and that of my students. The article discussed the collection of inspirational and reference documents, books and items kept by professional designers, with some illustrated examples, but highlighted the difference between the early days of modern design and now.
“There was a time, not long past, when seeking inspiration and influence was the challenge, not sorting it.”
The article goes on to note the difference between sharing a dog-eared copy of a design magazine, going to the library (which the authors define, tongue in cheek) and having to copy printed resources.
Of course, our students have gone through the same change. I had to go to the library,as a student, if I couldn’t buy the book. Locating was the problem. These days my students have to be able to classify, sift and order because there is so much information to hand. Of course, no all sources are equal and the implicit authority granted by physical publication now faces off against the ease of availability of the top ten hits on Google.
I find this more challenging in some ways but far more interesting in others. Yes, it’s easy to incorporate work that’s not your own but we are living in the data deluge – this fact allows us to have discussions about assessing quality, determining validity and authority and what plagiarism is and, importantly, how to avoid doing it, even unintentionally.
I much prefer a world where the problem is sifting. It forces us to look at far more interesting and important questions than “when does the library open” and allows us to spend our effort on understanding and using knowledge, without expending the majority of our effort on trying to locate already-published information. That whole “shoulders of giants” thing works a lot better when you can see the giants and find their shoulders! A surfeit of shoulders is a much better challenge to have.
From Zero To Legend (of Zelda): Rapid Achievements in Programming
Posted: March 13, 2012 Filed under: Education | Tags: Bret Victor, codea, education, higher education, resources, teaching, teaching approaches, tools Leave a commentWhen I first went to college, I had some programming experience because I’d been a keen Apple II programmer. This meant that I did have some idea of how much work was involved in making cool things happen on the computer. When you only had 7 colours in high res mode and you had to super pixel to make orange – you quickly realised that cool games took time. The people I started college with came in three varieties: people who could program in some sort of computer language (to varying degrees), people who knew what computers were but only as users, and people who had never touched one before. The people who had used them before, but had never programmed, used to gripe that we were doing all sorts of boring stuff and why couldn’t they just write a game? In fact, where were the graphics? (We were using a mainframe with dumb (text) terminals attached to it. GTC Infoton, VT102 emulation, 80×24, 2400 baud, all hooked up to an VAX 11/780. Yow.)
FEAR MY SERIAL NATURE!
These days, everyone has at least seen a computer and, as we all know, the things you can see on a computer today are mind-blowing. The graphics in just about any game can make you believe you’re in another world.
That, of course, is why a lot of people come to Computer Science and that, sadly, is also the reason that a lot of people are unhappy in first year. We teach them a lot of programming but, until recently, we didn’t use a language or environment that would allow them to do two important things:
- Do cool things.
- Show them off.
That’s where some of the great new(ish) languages are making their mark in early programming. Scratch and Alice, which I’ve referred to before, allow school students to make things happen. This allows them to do cool things and they can show it off to their friends – this makes what we do interesting, engaging, exciting and something that they may wish to pursue. At the other end of the spectrum, iOS and Android offer another pathway but they’re pretty high level in many ways – especially if you’re having to manually manage memory and write quite intricate code to handle Controllers or tie Views together. There’s no doubt that mobile platform computing is a short path to awesome but intro students may not be ready for the many complexities and pitfalls of starting in such a hostile world.
Students want to do amazing things as soon as they can because, for most of them, that’s why they came to us. Nobody came to listen to 7 lectures on the FOR loop. (At least, I really hope not!) We want to make amazing things happen.
So let me show you two other environments that may help students get there faster. One is an entire programming system based on the iPad, called Codea – your entire development environment sits on the iPad and you make programs, run programs and enjoy. Lots of good game widgets, premade content and you’re programming in Lua which is a pretty interesting language. (Full confession, Codea has been developed by some people I’ve known for a while but it’s interesting enough just to look at and, no, they don’t pay me. 🙂 ) It used to be called Codify (which explains the name in the video) but it’s now called Codea.
The other is from a link that a student sent me – he thought I’d find it interesting. (The full video is here.) It shows a development environment that allows you to modify variables in running code to fine-tune your game, allows you to freeze, roll forward and back and basically have fun while you program. There are also a lot of great concepts in what he discusses. The author and presenter is Bret Victor and he’s displaying his amazing interactive editor. (Seriously, go to Bret’s site. It will be one of the most interesting things you do all day.) I’ve put the intro video below. In the video, Bret’s editor is using JavaScript (another good “zero to legend” programming language) but with a lot of useful support that makes debugging and fine-tuning a lot more fun.
There are so many ways of approaching this problem – what’s great is that so many people are approaching it! They understand that everyone wants to be able to turn their dreams into reality and good support and development environments can help this.
[Edit: I’ve just realised that one of Mark’s posts that I had on my screen to read when I got back from SIGCSE provided another , and more detailed, look at Bret Victor. Seriously, if you’re not reading Mark’s Computer Science Education blog yet, you should. Save me the embarrassment of accidentally double posting. 🙂 ]
Nick Falkner 