Time Banking IV: The Role of the Oracle

Posted: June 14, 2012 | Author: | Filed under: Education | Tags: , , , , , , , , , , , , , | 1 Comment

I’ve never really gone into much detail on how I would make a system like Time Banking work. If a student can meet my requirements and submit their work early then, obviously, I have to provide some sort of mechanism that allows the students to know that my requirements have been met. The first option is that I mark everything as it comes in and then give the student their mark, allowing them to resubmit until they get 100%.

That’s not going to work, unfortunately, as, like so many people, I don’t have the time to mark every student’s assignment over and over again. I wait until all assignments have been submitted, review them as a group, mark them as a group and get the best use out of staying in the same contextual framework and working on the same assignments. If I took a piecemeal approach to marking, it would take me longer and, especially if the student still had some work to do, I could end up marking the same assignment 3,4, however many times and multiplying my load in an unsupportable way.

Now, of course I can come up with simple measures that the students can check for themselves. Of course, the problem we have here is setting something that a student can mis-measure as easily as they measure. If I say  “You must have at least three pages for an essay” I risk getting three pages of rubbish or triple spaced 18 point print. It’s the same for any measure of quantity (number of words, number of citations, length of comments and so on) instead of quality. The problem is, once again, that if the students were capable of determining the quality of their own work and determining the effort and quality required to pass, they wouldn’t need time banking because their processes are already mature!

So I’m looking for an indicator of quality that a student can use to check their work and that costs me only (at most) a small amount of effort. In Computer Science, I can ask the students to test their work against a set of known inputs and then running their program to see what outputs we get. There is then the immediate problem of students hacking their code and just throwing it against the testing suite to see if they can fluke their way to a solution. So, even when I have an idea of how my oracle, my measure of meeting requirements, is going to work, there are still many implementation details to sort out.

Fortunately, to help me, I have over five years worth of student data through our automated assignment submission gateway where some assignments have an oracle, some have a detailed oracle, some have a limited oracle and some just say “Thanks for your submission.” The next stage in the design of the oracle is to go back and to see what impact the indications of progress and completeness had on the students. Most importantly, for me, is the indication of how many marks a student had to get in order to stop trying to make fresh submissions. If before the due date, did they always strive for 100%? If late, did they tend to stop at more than 50% of achieved marks, or more than 40% in the case of trying to avoid receiving a failing grade based on low assignment submission?

Are there significant and measurable differences between assignments with an oracle and those that have none (or a ‘stub’, so to speak)? I know what many people expect to find in the data, but now I have the data and I can go and interrogate that!

Every time that I have questions like this about the implementation, I have a large advantage in that I already have a large control body of data, before any attempts were made to introduce time banking. I can look at this to see what student behaviour is like and try to extract these elements and use them to assist students in smoothing out their application of effort and develop more mature time management approaches.

Now to see what the data actually says – I hope to post more on this particular aspect in the next week or so.

Time Banking II: We Are a Team

Posted: June 12, 2012 | Author: | Filed under: Education | Tags: , , , , , , , , , , , , , , , | Leave a comment

In between getting my camera ready copy together for ICER, and I’m still pumped that our paper got into ICER, I’ve been delving deep into the literature and the psychological and pedagogical background that I need to confirm before I go too much further with Time Banking. (I first mentioned this concept here. The term is already used in a general sense to talk about an exchange of services based on time as a currency. I use it here within the framework of student assignment submission.) I’m not just reading in CS Ed, of course, but across Ed, sociology, psychology and just about anywhere else where people have started to consider time as a manageable or tradable asset. I thought I’d take this post to outline some of the most important concepts behind it and provide some rationale for decisions that have already been made. I’ve already posted the guidelines for this, which can be distilled down to “not all events can be banked”, “additional load must be low”, “pragmatic limits apply”, “bad (cheating or gaming) behaviour is actively discouraged” and “it must integrate with our existing systems”.

Time/Bank currency design by Lawrence Weiner. Photo by Julieta Aranda. (Question for Nick – do I need something like this for my students?)

Our goal, of course, is to get students to think about their time management in a more holistic fashion and to start thinking about their future activities sometime sooner the 24 hours before the due date. Rather than students being receivers and storers of deadline, can we allow them to construct their own timelines, within a set of limits? (Ben-Ari, 1998, “Constructivism in Computer Science Education”, SIGCSE,  although Ben-Ari referred to knowledge in this context and I’m adapting it to a knowledge of temporal requirements, which depends upon a mature assessment of the work involved and a sound knowledge of your own skill level.) The model that I am working with is effectively a team-based model, drawing on Dickinson and McIntyre’s 1997 work “Team Performance Assessment and Measurement: Theory, Methods and Applications.”, but where the team consists of a given student, my marking team and me. Ultimately our product is the submitted artefact and we are all trying to facilitate its timely production, but if I want students to be constructive and participative, rather than merely compliant and receptive, I have to involve them in the process. Dickinson and McIntyre identified seven roles in their model: orientation, leadership, monitoring, feedback, back-up (assisting/supporting), coordination and communication. Some of these roles are obviously mine, as the lecturer, such as orientation (establishing norms and keeping the group cohesive) and monitoring (observing performance and recognising correct contribution). However, a number of these can easily be shared between lecturer and student, although we must be clear as to who holds each role at a given time. In particular, if I hold onto deadlines and make them completely immutable then I have take the coordination role and handed over a very small fragment of that to the student. By holding onto that authority, whether it makes sense or not, I’m forcing the student into an authority-dependent mode.

(We could, of course, get into quite a discussion as to whether the benefit is primarily Piagiatien because we are connecting new experiences with established ideas, or Vygotskian because of the contact with the More Knowledgable Other and time spent in the Zone of Proximal Development. Let’s just say that either approach supports the importance of me working with a student in a more fluid and interactive manner than a more rigid and authoritarian relationship.)

Yes, I know, some deadlines are actually fixed and I accept that. I’m not saying that we abandon all deadlines or notion of immutability. What I am, however, saying is that we want our students to function in working teams, to collaborate, to produce good work, to know when to work harder earlier to make it easier for themselves later on. Rather than give them a tiny sandpit in which to play, I propose that we give them a larger space to work with. It’s still a space with edges, limits, defined acceptable behaviour – our monitoring and feedback roles are one of our most important contributions to our students after all – but it is a space in which a student can have more freedom of action and, for certain roles including coordination, start to construct their own successful framework for achievement.

Much as reading Vygotsky gives you useful information and theoretical background, without necessarily telling you how to teach, reading through all of these ideas doesn’t immediately give me a fully-formed implementation. This is why the guidelines were the first things I developed once I had some grip on the ideas, because I needed to place some pragmatic limits that would allow me to think about this within a teaching framework.  The goal is to get students to use the process to improve their time management and process awareness and we need to set limits on possible behaviour to make sure that they are meeting the goal. “Hacks” to their own production process, such as those that allow them to legitimately reduce their development time (such as starting the work early, or going through an early prototype design) are the point of the exercise. “Hacks” that allow them to artificially generate extra hours in the time bank are not the point at all. So this places a requirement on the design to be robust and not susceptible to gaming, and on the orientation, leadership and monitoring roles as practiced by me and my staff. But it also requires the participants to enter into the spirit of it or choose not to participate, rather than attempting to undermine it or act to spite it.

The spontaneous generation of hours was something that I really wanted to avoid. When I sketched out my first solution, I realised that I had made the system far too complex by granting time credits immediately, when a ‘qualifying’ submission was made, and that later submissions required retraction of the original grant, followed by a subsequent addition operation. In fact, I had set up a potential race condition that made it much more difficult to guarantee that a student was using genuine extension credit time. The current solution? Students don’t get credit added to their account until a fixed point has passed, beyond which no further submissions can take place. This was the first of the pragmatic limits – there does exist a ‘no more submissions’ point but we are relatively elastic to that point. (It also stops students trying to use obtained credit for assignment X to try and hand up an improved version of X after the due date. We’re not being picky here but this isn’t the behaviour we want – we want students to think more than a week in advance because that is the skill that, if practised correctly, will really improve their time management.)

My first and my most immediate concern was that students may adapt to this ‘last hand-in barrier’ but our collected data doesn’t support this hypothesis, although there are some concerning subgroups that we are currently tearing apart to see if we can get more evidence on the small group of students who do seem to go to a final marks barrier that occurs after the main submission date.

I hope to write more on this over the next few days, discussing in more detail my support for requiring a ‘no more submissions’ point at all. As always, discussion is very welcome!

Let’s Turn Down the Stupid (Ignorance is Our Enemy)

Posted: June 11, 2012 | Author: | Filed under: Education, Opinion | Tags: , , , , , , | 1 Comment

(This is a longish opinion piece that has very little educational discussion. I leave it you as to whether you wish to read it or not.)

I realise that a number of you may read my blog posts and think “Well, how nice for him. He has tenure in a ‘good’ University, has none of his own kids to worry about and is obviously socially mobile and affluent.” Some of you may even have looked up my public record salary when I talk about underpaying teachers and wondered why I don’t just shut up and enjoy my life, rather than blathering on here. It would be easy to cast me as some kind of Mr Happy/Pollyanna figure, always seeing the positive and rushing out onto the sports field with a rousing “We’re all winners, children” attitude.

Nothing could be further from the truth. I get up every day knowing that the chances are that I will not make a difference, that all of my work will be undone by a scare campaign in a newspaper, that I may catch a completely preventable disease because too few people got vaccinated, that I and my family may not have enough food or lose my house because people ignore science, that anti-scientific behaviour is clawing back many of the victories that we have already achieved.

I’m no Pollyanna. I get up every day ready to fight ignorance and try to bring knowledge to places where ignorance reigns. Sometimes I manage it – those are good days. But I can’t just talk to my own students, I have to reach out into the community because I see such a small percentage of a small percentage as my students. If I want lasting change, and I believe that most educators are all trying to change the world for the better, then I have to deal with the fact that my message, and my students, have to be able to be seen outside of our very small and isolated community.

This morning, while out running, we had gone a bit over 14 kilometres (about 9 miles) when I saw a cyclist up ahead off us, stopped on a little wooden ramp that went under one of the bridges. He heard us coming and waved us down, very quickly.

Someone had strung fishing line across the path, carefully anchored on both sides, at around mid-chest height for adult runners and walkers, or neck/head height for children.

Of course, the moment we realised this we looked around for the utter idiots who were no doubt waiting to film this or watch it but they showed a modicum of sense in that we couldn’t see them. (Of course, what could we have done even if we had seen them. They were most likely children and the police aren’t likely to get involved for a ‘fishing line’ related incident.) What irritated me most about this was that I was running with someone who was worried about the future and I was solemnly telling her that I had great hope for the future, that the problems could be solved if we worked at it and this is what I always tried to get across to my students.

And then we nearly got garrotted by an utterly thoughtless act of stupidity. Even a second’s thought would lead you to the understanding that this was more than a joke, it was potentially deadly. And yet, the people who put this up, who I have no doubt waited to watch or film it, were incapable of doing this. I can only hope that they were too young, or too mentally incapacitated, to know better. Because when someone knowingly does this, it takes them from ignorance to evil. Fortunately, the number of truly evil people, people who do these things in full knowledge and delight, are small. At least, that’s what I tell myself to get myself to sleep at night. We must always be watchful for evil but in the same way that we watch for the infrequently bad storm – when we see the signs, we batten down, but we don’t live our lives in the storm cellar. Ignorance, for me, is far more prevalent and influential than evil – and often has very similar effects as it can take people from us, by killing or injuring them or by placing them into so much mental or physical pain that they can no longer do what they could have done with their lives.

The biggest obstacle we face is ignorance and acts taken in ignorance, whether accidentally or wilfully so. There’s no point me training up the greatest mind in the history of the world, only for that person to be killed by someone throwing a rock off a bridge for fun. Today, I could easily have been seriously injured because someone thought it was funny to watch people run into an obstacle at speed. Yes, the line probably would have broken and I was unlikely to have suffered too much harm. Unless it didn’t. Unless it took out an eye.

But I’m not giving up. I say, mostly joking, when I run across things like this “This is why we fight.” and I mean it. This is exactly why education is important. This is why teachers are important. This is why knowledge is important. Because, without all of these, ignorance will win and it will eventually kill us.

I am sick of stupid, ignorant and evil people. I’m sick of grown men getting away with disgraceful behaviour because “boys will be boys”. I’m sick of any ignorant or thoughtless act being tolerated with “Oh well, these things happen”. However, me being sick of this does nothing unless I act to stop it. Me acting to stop it may do nothing. Me doing nothing to stop it definitely does nothing.

What are the Fiction and Non-Fiction Equivalents of Computer Science?

Posted: June 9, 2012 | Author: | Filed under: Education, Opinion | Tags: , , , , , , , , , , , , , , | 2 Comments

I commented yesterday that I wanted to talk about something covered in Mark’s blog, namely if it was possible to create an analogy between Common Core standards in different disciplines with English Language Arts and CS as the two exemplars. In particular, Mark pondered, and I quote him verbatim:

”Students should read as much nonfiction as fiction.”  What does that mean in terms of the notations of computing? Students should read as many program proofs as programs?  Students should read as much code as comments?

This a great question and I’m not sure that I have much of an answer but I’ve been enjoying thinking about it. We bandy the terms syntax and semantics around in Computer Science a lot: the legal structures of the programs we write and the meanings of the components and the programs. Is it even meaningful to talk about fiction and non-fiction in these terms and where do these fit? I’ve gone in a slightly different direction from Mark but I hope to bring it back to his suggestions later on.

I’m not an English specialist, so please forgive me or provide constructive guidance as you need to, but both fiction and non-fiction rely upon the same syntactic elements and the same semantic elements in linguistic terms – so the fact that we must have legal programs with well-defined syntax and semantics pose no obstacle to a fictional/non-fictional interpretation.

Forgive me as I go to Wikipedia for definitions for fiction and non-fiction for a moment:

“Non-fiction (or nonfiction) is the form of any narrative, account, or other communicative work whose assertions and descriptions are understood to be factual.” (Warning, embedded Wikipedia links)

“Fiction is the form of any narrative or informative work that deals, in part or in whole, with information or events that are not factual, but rather, imaginary—that is, invented by the author” (Again, beware Wikipedia).

Now here we can start to see something that we can get our teeth into. Many computer programs model reality and are computerised representation of concrete systems, while others may have no physical analogue at all or model a system that has never or may never exist. Are our simulations and emulations of large-scale system non-fiction? If so, is a virtual reality fictional because it has never existed or non-fictional because we are simulating realistic gravity? (But, of course, fiction is often written in a real world setting but with imaginary elements.)

From a software engineering perspective, I can see an advantage to making statements regarding abstract representations and concrete analogues, much as I can see a separation in graphics and game design between narrative/event engine construction and the physics engine underneath.

Is this enough of a separation? Mark’s comments on proof versus program is an interesting one: if we had an idea (an author’s creation) then it is a fiction until we can determine that it exists, but proof or implementation provides this proof of existence. In my mind, a proof and a program are both non-fiction in terms of their reification, but the idea that they span may still be fictional. Comments versus code is also very interesting – comments do not change the behaviour of code but explain, from the author’s mind, what has happened. (Given some student code and comment combinations, I can happily see a code as non-fiction, comment as fiction modality – or even comment as magical reality!)

Of course, this is all an enjoyable mental exercise, but what can I take from this and use in my teaching. Is there a particular set of code or comments that students should read for maximum benefit and can we make a separation that, even if not partitioned so neatly across two sets, gives us the idea of what constitutes a balanced diet of the products of our discipline?

I’d love to see some discussion on this but, if nothing here, then I’m happy to buy the first round of drinks at HERDSA or ICER to start a really good conversation going!

Eating Your Own Dog Food (How Can I Get Better at Words with Friends?)

Posted: June 4, 2012 | Author: | Filed under: Education | Tags: , , , , , , , , , , , , , , , , | 1 Comment

I am currently being simultaneously beaten in four games of Words with Friends. This amuses me far more than it bugs me because it appears that, despite having a large vocabulary, a (I’m told) quick wit and being relatively skilled in the right word in the right place – I am rather bad at a game that should reward at least some of these skills.

One of the things that I dislike, and I know that my students dislike, is when someone stands up and says “To solve problem X, you need to take set of actions S.” Then, when you come to X, or you find that person’s version of a solution to X, it’s not actually S that is used. It’s “S-like” or “S-lite” or “Z, which looks like an S backwards and sounds like it if you’re an American with a lisp.”

There’s a term I love called “eating your own dog food” (Wikipedia link) that means that a company uses the products that it creates in order to solve the problems for which a customer would buy their products. It’s a fairly simple mantra: if you’re making the best thing to solve Problem X, then you should be using it yourself when you run across Problem X. Now,of course, a company can do this by banning or proscribing any other products but this misses the point. At it’s heart, dogfooding means that, in a situation where you are free to choose, you make a product so good that you would choose it anyway.

It speaks to authenticity when you talk about your product and it provides both goals and thinking framework. The same thing works for education – if I tell someone to take a certain approach to solve a problem, then it should be one that I would use as well.

So, if a student said to me “I am bad at this type of problem,” I’d start talking to them to find out exactly what they’re good and bad at, get them to analyse their own process, get them to identify some improvement strategies (with my guidance and suggestions) and then put something together to get it going. Then we’d follow up, discuss what happened, and (with some careful scaffolding) we’d iteratively improve this as far as we could. I’d also be open to the student working out whether the problem is actually one that they need to solve – although it’s a given that I’ll have a strong opinion if it’s something important.

So, let me eat my own dog food for this post, to help me get better at Words with Friends, to again expose my thinking processes but also to demonstrate the efficacy of doing this!

Step 1: What’s the problem?

So, I can get reasonable scores at Words with Friends but I don’t seem to be winning. Words with Friends is a game that rewards you for playing words with “high value” tiles on key positions that add score multipliers. The words QATS can be worth 13 or 99 depending on where it is placed. You have 7 randomly selected tiles with different letters, and a range of values for letters in a 1:1 association, but must follow strict placement and connection rules. In summary, a Words with Friends game is a connected set of tiles, where each set of tiles placed must form a valid word once set placement is complete, and points are calculated from the composition and placement of the tiles, but bonus spaces on the board only count once. The random allocation of letters means that you have to have a set of strategies to minimise the negative impact of a bad draw and to maximise the benefit of a good draw. So you need a way of determining the possible moves and then picking the best one.

Some simple guidelines that help you to choose words can be formed along the lines of the number of base points by letter (so words featuring Q, X or J will be worth more because these are high value letters), the values of words will tend to increase as the word length increases as there are more letters with values to count (although certain high value letters cannot be juxtaposed – QXJJXWY is not a word, sadly), but both of these metrics are overshadowed by the strategic placement of letters to either extend existing words (allowing you to recount existing tiles and extending point 2) or to access the bonus spaces. Given that QATS can be worth 99 points as a four-letter word if played in the right place, it might be worth ignoring QUEUES earlier if think you can reach that spot.

Step 2: So where is my problem?

After thinking about my game, I realised that I wasn’t playing Words with Friends properly, because I wasn’t giving enough thought to the adversarial nature of the occupancy of the bonus spaces. My original game was more along the lines of “look at letters, look at board, find a good word, play it.” As a result, any occupancy of the bonus spaces was a nice-to-have, rather than a must-have. I also didn’t target placement that allowed me to count tiles already on the board and, looking at other games, my game is a loose grid compared to the tight mesh that can earn very large points.

I’m also wasn’t thinking about the problem space correctly. There are a fixed number of tiles in the game, with known distribution. As tiles are played, I know how many tiles are left and that up to 14 of them are in my and my opponent’s hand. If I know how many tiles there are of each letter, I can play with a reasonable idea of the likelihood of my opponent’s best move. Early on, this is hard, but that’s ok, because we can both play in a way that doesn’t give a bonus tile advantage. Later on, it’s probably more useful.

Finally, I was trying to use words that I knew, rather than words that are legal in Words with Friends. I had no idea that the following were acceptable until (at least once out of desperation) I tried them. Here are some you might (nor might now) know: AA, QAT, ZEE, ZAS, SCARP, DYNE. The last one is interesting, because it’s a unit of force, but BRIX, a unit used to measure concentration (often of sugar) isn’t a legal word.

So, I had three problems, most of which relate to the fact that I’m more used to playing “Take 2” (a game played with Scrabble tiles but no bonus spaces) than “Scrabble” itself, where the bonus spaces are crucial.

Step 3: What are the strategies for improvements?

The first, and most obvious, strategy is to get used to playing in the adversarial space and pay much closer attention to which bonus spaces I leave open in my play and to increase my recounting of existing tiles. The second is to start keeping track of tiles that are out and play to the more likely outcome. Finally, I need to get a list of which words are legal in Words with Friends and, basically, learn them.

Step 4: Early outcomes

After getting thrashed in my first games, I started applying the first strategy. I have since achieved words worth over 100 points and, despite not winning, the gap is diminishing. So this appears to be working.

The second and the third… look, it’s going to sound funny but this seems like a lot of work for a game. I quite like playing the best word I can think of without having to constrain myself to play some word I’m never going to actually use (when we’re up to our elbows in aa, I will accept your criticism then) or sit there eliminating tiles one-by-one (or using an assistant to do it). Given that I’m not even sure that this is the way people actually play, I’m probably better off playing a lot of games and naturally picking up words that occur, rather than trying to learn them all in one go.

Of course, if a student said something along the last lines to me, then they’re saying that they don’t mind not succeeding. In this case, it’s perfectly true. I enjoy playing and, right now, I don’t need to win to enjoy the game.

Just as well really, I think I’m about to lose four games within a minute of each other. That’s four in a row – pity, if there were three of them I could do a syzygy joke.

Step 5: Discussion and Iteration

So, here’s the discussion and my chance to think about whether my strategies need modification to achieve my original goal. Now, if I keep that goal at winning, then I do need to keep iterating but I have noticed that with a simple change of aiming more a the bonuses, I get a good “Yeah” from a high points word that probably won’t be matched by winning a game.

To wrap up, having looked at the problem, thought through it and make some constructive suggestions regarding improvement, I’ve not only improved my game but I’ve improved my understanding and enjoyment of the activity. I feel far more in control of my hideous performance and can now talk to more people about other ways to improve that maintain that enjoyment.

Now, of course, I imagine that a million WwF players are going to jump in and say “nooooo! here’s how you do it.” Please do so! Right now I’m talking to myself but I’d love some guidance for iterative improvement.

Learning from other people – Academic Summer Camp (except in winter???)

Posted: June 3, 2012 | Author: | Filed under: Education | Tags: , , , , , , , , , , , , , , , | Leave a comment

I’ve just signed up for the Digital Humanities Winter Institute course on “Large-scale text analysis with R”. K read about it on ProfHacker and passed it on to me thinking I’d be interested. Of course, I was, but it goes well beyond learning R itself. R is a statistically focused programming package that is available for free for most platforms. It’s the statistical (and free, did I mention that?) cousin to the mathematically inclined Matlab.

I’ve spoken about R before and I’ve done a bit of work in it but, and here’s why I’m going, I’ve done all of it from within a heavily quantitative Computer Science framework. What excites me about this course is that I will be working with people from a completely different spectrum and with a set of text analyses with which I’m not very familiar at all. Let me post the text of the course here (from this website) [my bold]:

Large-Scale Text Analysis with R
Instructor: Matt Jockers, Assistant Professor of Digital Humanities, Department of English, University of Nebraska, Lincoln

Text collections such as the HathiTrust Digital Library and Google Books have provided scholars in many fields with convenient access to their materials in digital form, but text analysis at the scale of millions or billions of words still requires the use of tools and methods that may initially seem complex or esoteric to researchers in the humanities. Large-Scale Text Analysis with R will provide a practical introduction to a range of text analysis tools and methods. The course will include units on data extraction, stylistic analysis, authorship attribution, genre detection, gender detection, unsupervised clustering, supervised classification, topic modeling, and sentiment analysis. The main computing environment for the course will be R, “the open source programming language and software environment for statistical computing and graphics.” While no programming experience is required, students should have basic computer skills and be familiar with their computer’s file system and comfortable with the command line. The course will cover best practices in data gathering and preparation, as well as addressing some of the theoretical questions that arise when employing a quantitative methodology for the study of literature. Participants will be given a “sample corpus” to use in class exercises, but some class time will be available for independent work and participants are encouraged to bring their own text corpora and research questions so they may apply their newly learned skills to projects of their own.

There are two things I like about this: firstly that I will be exposed to such a different type and approach to analysis that is going to be immediately useful in the corpus analyses that we’re planning to carry out on our own corpora, but, secondly, because I will have an intensive dedicated block of time in which to pursue it. January is often a time to take leave (as it’s Summer in Australia) – instead, I’ll be rugged up in the Maryland chill, sitting with like-minded people and indulging myself in data analysis and learning, learning, learning, to bring knowledge home for my own students and my research group.

So, this is my Summer Camp. My time to really indulge myself in my coding and just hack away at analyses and see what happens.

I’ve also signed up to a group who are going to work on the “Million Syllabi Project Hack-a-thon“, where “we explore new ways of using the million syllabi dataset gathered by Dan Cohen’s Syllabus Finder Tool” (from the web site). 10 years worth of syllabi to explore, at a time when my school is looking for ways to be able to teach into more areas, to meet more needs, to create a clear and attractive identity for our discipline? A community of hackers looking at ways of recomposing, reinterpreting and understanding what is in this corpus?

How can I not go? I hope to see some of you there! I’ll be the one who sounds Australian and shivers a lot.

It is an amazing day.

Posted: June 2, 2012 | Author: | Filed under: Education | Tags: , , , , , , , , , , , | Leave a comment

Today marks the start of my sixth month of blogging. It is also the day after my largest number of hits, my most ‘viewed’ month and my my most ‘viewed’ week. Thank you to everyone who has visited and all of those of you who have taken the time to comment! It is amazing, in many ways, how ordinary a day this is to me, given how much is going on. But, of course, every day is amazing because every day is a new day. There is always the chance to do something new, something different, something wonderful.

Someone else thought that this was what an amazing day looked like. I agree!

I have tried to share my own progress in terms of understanding key concepts of learning and teaching, as a student, as a practitioner, and as a person. I can only hope that some of the people who have stumbled across this blog have found something useful here. (Sorry to the people who were looking for Page 3 girls.)

Analysing the searches that brought people here has, as I’ve previously noted, been somewhat sad as “alone in the crowd” is still the biggest draw. I worry because if you are looking for words to live by, this may not be the place to find the words that keep you alive. So, if your searches bring you here as well, I can only hope that you find what you need.

The word ‘hope’ springs up a lot in my writing. It’s a standard English form (I hope that you are well) but, for me, it is more than that. I have a great hope for the future – I would have difficulty doing my job if I did not. Every semester, I get to see a new group of students, some of whom I may know, and we start again. Knowledge, learning, hope.

For no other reason than that, the hope of something better, the hope of something brighter, and the hope that I may be helping to illuminate the way ahead – it is an amazing day.

The Confusing Message: Sourcing Student Feedback

Posted: May 26, 2012 | Author: | Filed under: Education | Tags: , , , , , , , , , , , , , | Leave a comment

Once, for a course which we shall label ‘an introduction to X and Y’, I saw some feedback from a student that went as follows. A single student, on the same feedback form, and in adjacent text boxes, gave these answers:

What do you like most about this course: the X

What would you like to see happen to improve the course: less X, more Y!

Now, of course, this not inherently contradictory but, honestly, it’s really hard to get the message here. You think that X is great but less useful than Y, although you like X more? You’re a secret masochist and you like to remove pleasure from your life?

As (almost) always, the problem here is that we these two questions, asked in adjacent text boxes, are asking completely different things. Survey construction is an art, a dark and mysterious art, and a well-constructed survey will probably not answer a question once, in one way. It will ask the same question in multiple ways, sometimes in the negative, to see if the “X” and “not ( not (X))” scores line up for each area of interest. This, of course, assumes that you have people who are willing to fill out long surveys and give you reliable answers. This is a big assumption. Most of the surveys that I work with have to fit into short time frames and are Likert-based with text boxes. Not quite yes/no tick/flick but not much more and very little opportunity for mutually interacting questions.

The Red Tick of Courage!

Our student experience surveys are about 10 questions long with two text boxes and are about the length that we can fit into the end of a lecture and have the majority of students fill out and return. From experience, if I construct larger surveys, or have special ‘survey-only’ sessions, I get poor participation. (Hey, I might just be doing it wrong. Tips and help in the comments, please!)

Of course, being Mr Measurement, I often measure things as side effects of the main activity. Today, I held a quiz in class and while everyone was writing away, I was actually getting a count of attendees because they were about to hand up cards for marking. This gives me an indicator of attendance and, as it happens, two weeks away from the end of the course, we’re still getting good attendance. (So, I’m happy.) I can also see how the students are doing with fundamental concepts so I can monitor that too.

I’m fascinated by what students think about their experience but I need to know what they need based on their performance, so that I can improve their performance without having to work out what they mean. The original example would give me no real insight into what to do and how to improve – so I can’t really do anything with any certainty. If the student had said “I love X but I feel that we spent too much time on it and it could be just as good with a little less.” then I know what I can do.

I also sometimes just ask for direct feedback in assignments, or in class, because then I’ll get the things that are really bugging or exciting people. That also gives me the ability to adapt to what I hear and ask more directed questions.

Student opinion and feedback can be a vital indicator of our teaching efficacy, assuming that we can find out what people think rather than just getting some short and glib answers to questions that don’t really probe in the right ways, where we never get a real indication of their thoughts. To do this requires us to form a relationship, to monitor, to show the value of feedback and to listen. Sadly, that takes a lot more work than throwing out a standard form once a semester, so it’s not surprising that it’s occasionally overlooked.

Proscription and Prescription: Bitter Medicine for Teachers

Posted: May 24, 2012 | Author: | Filed under: Education | Tags: , , , , , , , , , , , , , , , , | Leave a comment

Australia is a big country. A very big country. Despite being the size of the continental USA, it has only 22,000,000 people, scattered across the country and concentrated in large cities. This allows for a great deal of regional variation in terms of local culture, accents (yes, there is more than one Australian accent) and local industry requirements. Because of this, despite having national educational standards and shared ideas of what constitutes acceptable entry levels for University, there are understandable regional differences in the primary, secondary and tertiary studies.

Maintaining standards is hard, especially when you start to consider regional issues – whose standards are you maintaining. How do you set these standards? Are they prescriptions (a list of things that you must do) or proscriptions (a list of things that you mustn’t do)? There’s a big difference in course and program definition depending upon how you do this. If you prescribe a set textbook then everyone has to use it to teach with but can bring in other materials. If you proscribe unauthorised textbooks then you have suddenly reduced the amount of initiative and independence that can be displayed by your staff.

Excuse me, Doctor, you appear to be writing in invisible ink…

As always, I’m going to draw an analogue with our students to think about how we guide them. Do we tell them what we want and identify those aspects that we want them to use, or do we tell them what not to do, limit their options and then look surprised when they don’t explore the space and hand in something that conforms in a dull and lifeless manner?

I’m a big fan of combining prescription, in terms of desirable characteristics, and proscription, in terms of pitfalls and traps, but in an oversight model that presents the desirable aspects first and monitors the situation to see if behaviour is straying towards the proscribed. Having said that, the frequent flyers of the proscription world, plagiarism and cheating, always get mentioned up front – but as the weak twin of the appropriate techniques of independent research, thoughtful summarisation, correct attribution and doing your own work. Rather than just saying “DO NOT CHEAT”, I try to frame it in terms of what the correct behaviour is and how we classify it if someone goes off that path.

However, any compulsory inclusions or unarguable exclusions must be justified for the situation at hand – and should be both defensible and absolutely necessary. When we start looking at a higher level, above the individual school to the district, to the region, to the state, to the country, any complex set of prescriptions and proscriptions is very likely to start causing regional problems. Why? Because not all regions are the same. Because not all districts have the money to meet your prescriptions. Because not all cultures may agree with your proscriptions.

This post was triggered by a post from a great teacher I know, to whom I am also related, who talked about having to take everything unofficial out of her class. Her frustration with this, the way it made her feel, the way it would restrict her – an award winning teacher – made me realise how privileged I am to work in a place where nobody really ever tells me what to do or how to teach. While it’s good for me to remember that I am privileged in this regard, perhaps it’s also good to think about the constant clash between state, bureaucracy and education that exist in some other places.

If We’re Going To Measure, Let’s Measure Properly: Teaching Isn’t a NASCAR Race

Posted: May 22, 2012 | Author: | Filed under: Education | Tags: , , , , , , , , , , | 1 Comment

I’ve been reading a Huff Post piece on teacher assessment, entitled “Carolyn Abbott, The Worst 8th Grade Math Teacher In New York City, Victim Of Her Own Success”, where a teacher, Carolyn Abbot, at a gifted and talented school in Manhattan was rated being the worst teacher in 8th grade.

The problem, it appears, is the measurement used where your contribution is based upon whether your students have performed better or worse than last year on the Teacher Data Report, a measure used to assess contribution to English and Math. So here’s the problem. The teacher taught maths to grades 7 and 8 and her Grade 7 students achieved at the 98th percentile for their test in 2009. Therefore, according to the Teacher Data Reports modelling process, the same students should have achieved 97th percentile in their Grade 8 tests the following year. They only managed 89th percentile. Abbot had made a significant negative contribution to her students, by this logic, and her ranking was the lowest in NYC 8th grade mathematics teachers.

Yes, you read that right. She’s kept the students in the 1.5-2 standard deviations above the norm category. The students have moved up a year and are now starting to run into the puberty zone, always fun, they’re still scoring in the 89th percentile – and she’s the worst teacher in NYC. Her students struggle with the standardised testing itself: the non-mathematical nature of the tests, the requirement to put in a single answer when the real answer is potentially more complex, the fact that multiple choice can be trained for (rather than test anything) – and they’re still kicking out at the +85 level. Yet, she’s the worst 8th grade math teacher in NYC.

This also goes against one of my general principles of assessment, in that the performance of someone else affects the assessment of your performance. (Yes, that leaves me at odds with national testing schemes, because I don’t see a way that they can be meaningfully calibrated across many different teaching systems and economic influences. It’s obvious that New York haven’t worked it out properly for one system and one economy!) Having a notion of acceptable and unacceptable is useful here. Having a notion of exemplary, acceptable and unacceptable is useful here. Having a notion of best and worst is meaningless, because all these teachers could score 100/100 and one scores 100/99 and they’re the worst. Ranking must be combined with standards of acceptability where professional practice is required. This isn’t a NASCAR race: in teaching, everyone can cross the line in a way that they win.

I am a big fan of useful, carefully constructed and correctly used measurement but this story is an example of what happens if you come up with a simple measure that gives you a single number that isn’t much use but is used as if it means something. Now, if over time, you saw a large slide in scores from one teacher and that dropped down low enough, then maybe this number would mean something but any time that you simple number has to come with an explanation – it’s not that simple anymore.

In this case, what’s worse is that the rankings were published with names. Names of teachers and names of schools. Abbott’s boss reassured her that he would still put her up for tenure but felt he had to warn her that someone else might take these rankings into account.

Abbott’s ranking doesn’t matter to her much anymore, because this teacher has now left teaching and is undertaking a PhD in Mathematics instead. Great for us at University because having good teachers who then successfully complete PhDs often works out very well – they’re highly desirable employees in many ways. Not so good for the students at her school who have been deprived of a teacher who managed to get a group of kids to the 98th percentile on Grade 7 Math, providing a foundation that will probably be with them for their whole lives (even if we quibble and it’s down to the 89th) and giving them a better start for their academic future.

But that’s ok, kids, because she was the worst teacher you’d ever have. Oh, of course, there’s another new “worst” teacher because that’s how our ranking system works. Sorry about that. Good luck, Carolyn Abbott!

← Older Entries
Newer Entries →