Data: Harder to Anonymise Yourself Than You Might Think

There’s a lot of discussion around a government’s use of metadata at the moment, where instead of looking at the details of your personal data, government surveillance is limited to looking at the data associated with your personal data. In the world of phone calls, instead of taping the actual call, they can see the number you dialled, the call time and its duration, for example. CBS have done a fairly high-level (weekend-suitable) coverage of a Stanford study that quickly revealed a lot more about participants than they would have thought possible from just phone numbers and call times.

But how much can you tell about a person or an organisation without knowing the details? I’d like to show you a brief, but interesting, example. I write fiction and I’ve recently signed up to “The Submission Grinder“, which allows you to track your own submissions and, by crowdsourcing everyone’s success and failures, to also track how certain markets are performing in terms of acceptance, rejection and overall timeliness.

Now, I have access to no-one else’s data but my own (which is all of 5 data points) but I’ll show you how assembling these anonymous data results together allows me to have a fairly good stab at determining organisational structure and, in one case, a serious organisational transformation.

Let’s start by looking at a fairly quick turnover semi-pro magazine, Black Static. It’s a short fiction market with horror theming. Here’s their crowd-sourced submission graph for response times, where rejections are red and acceptances are green. (Sorry, Damien.)

Black Static - Response Time Graph

Black Static – Response Time Graph

Black Static has a web submission system and, as you can see, most rejections happen in the first 2-3 weeks. There is then a period where further work goes on. (It’s very important to note that this is a sample generated by those people who are using Submission Grinder, which is a subset of all people submitting to Black Static.) What this looks like, given that it is unlikely that anyone could read a lot 4,000-7,000 manuscripts in detail at a time, is that the editor is skimming the electronic slush pile to determine if it’s worth going to other readers. After this initial 2 week culling, what we are seeing is the result of further reading  so we’d probably guess that the readers’ reviews are being handled as they come in, with some indication that this is one roughly weekly – maybe as a weekend job? It’s hard to say because there’s not much data beyond 21 days so we’re guessing.

Let’s look at Black Static’s sister SF magazine, Interzone, now semi-pro but still very highly regarded.

Interzone - Response Times Graph

Interzone – Response Time Graph

Lots more data here! Again, there appears to be a fairly fast initial cut-off mechanism from skimming the web submission slush pile. (And I can back this up with actual data as Interzone rejected one of my stories in 24 hours.) Then there appears to be a two week period where some thinking or reading takes place and then there’s a second round of culling, which may be an editorial meeting or a fast reader assignment. Finally we see two more fortnightly culls as the readers bring back their reviews. I think there’s enough data here to indicate that Interzone’s editorial group consider materials most often every fortnight. Also the acceptances generated by positive reviews appear to be the same quantity as those from the editors – although there’s so little data here we’re really grabbing at tempting looking straws.

Now let’s look at two pro markets, starting with the Magazine of Fantasy & Science Fiction.

Fantasy & Science Fiction - Response Time Graph

Fantasy & Science Fiction – Response Time Graph

This doesn’t have the same initial culling process that the other two had, although it appears that there is a period of 7-14 days when a lot of work has been reviewed and then rejected – we don’t see as much work rejected again until the 35 day mark, when it looks like all reader reviews are back. Notably, there is a large gap between the initial bunch of acceptances (editor says ‘yes’) and then acceptances supported by reviewers. I’m speculating now but I wonder if what we’re seeing between that first and second group of acceptances are reviewers who write back in and say “Don’t bother” quickly, rather than assembling personalised feedback for something that could be salvaged. Either way, the message here is simple. If you survive the first four weeks in F&SF system, then you are much less likely to be rejected and, with any luck, this may translate (worse case) into personal suggestions for improvement.

F&SF has a postal submission system, which makes it far more likely that the underlying work is going to batched in some way, as responses have to go out via mail and doing this in a more organised fashion makes sense. This may explain why this is such a high level of response overall for the first 35 days, as you can’t easily click a button to send a response electronically and there’re a finite number of envelopes any one person wants to prepare on any given day. (I have no idea how right I am but this is what I’m limited to by only observing the metadata.)

Tor.com has a very interesting graph, which I’ll show below.

Tor.com - Response Time Graph

Tor.com – Response Time Graph

Tor.com pays very well and has an on-line submission system via e-mail. As a result, it is positively besieged with responses and their editorial team recently shut down new submissions for two months while they cleared backlog. What interested me in this data was the fact that the 150 day spike was roughly twice as high as the 90 and 120. Hmm – 90, 120, 150 as dominant spikes. Does that sound like a monthly editors’ meeting to anyone else? By looking at the recency graph (which shows activity relative to today) we can see that there has been an amazing flurry of activity at Tor.com in the past month. Tor.com has a five person editorial team (from their website) with reading and support from two people (plus occasional others).  It’s hard for five people to reach consensus without discussion so that monthly cycle looks about right. But it will take time for 7 people to read all of that workload, which explains the relative silence until 3 months have elapsed.

What about that spike at 150? It could be the end of the initial decisions and the start of “worth another look” pile so let’s see if their web page sheds any light on it. Aha!

Have you read my story? We reply to everything we’ve finished evaluating, so if you haven’t heard from us, the answer is “probably not.” At this point the vast majority of stories greater than four months old are in our second-look pile, and we respond to almost everything within seven months.

I also wonder if we are seeing previous data where it was taking longer to get decisions made – whether we are seeing two different time management strategies of Tor.com at the same time, being the 90+120 version as well as the 150 version. Looking at the website again.

Response times have improved quite a bit with the expansion of our first reader team (emphasis mine), and we now respond to the vast majority of stories within three months. But all of the stories they like must then be read by the senior editorial staff, who are all full-time editors with a lot on our plates.

So, yes, the size of Tor.com’s slush pile and the number of editors that must agree basically mean that people are putting time aside to make these decisions, now aiming at 90 days, with a bit of spillover. It looks like we are seeing two regimes at once.

All of this information is completely anonymous in terms of the stories, the authors and any actual submission or acceptance patterns that could relate data together. But, by looking at this metadata on the actual submissions, we can now start to get an understanding of the internal operations of an organisation, which in some cases we can then verify with publicly held information.

Now think about all the people you’ve phoned, the length of time that you called them and what could be inferred about your personal organisation from those facts alone. Have a good night’s sleep!

 


Should I go to Missouri? (#SIGCSE #SIGCSE2015)

The amazing PBL team (Raja, Zbsyzek, Ed and me) have been accepted to run a Puzzle-Based Learning workshop at SIGCSE 2015 in KC, Missouri. I was really excited about this until the recent news about Anita Sarkeesian in Utah broke and it suddenly occurred to me to check to see whether Missouri had concealed carry laws that applied the same way and whether the SIGCSE people had a policy to prevent guns being carried into the auditorium space.

As it turns out, open carry  (for handguns, not long weapons) is permitted in Missouri as of October 11, 2014, (yes, that’s 5 days ago) and these state laws override any local laws on open carry. Concealed carry is also an option but you have to have your permit with you at all times – not carrying the permit will attract a $35 fine! Goodness! I know that would put me off – that’s almost $40 in Australian currency and is nearly 20 minutes of consulting work.

From wikipedia, a gun locked out for safe transport.

From wikipedia, a gun locked out for safe transport.

Missouri also has no permit to purchase, no firearm registration, no owner licensing, no assault weapon law, no magazine capacity restriction and no restriction on “NFA weapons” – which means machine guns.

As far as I can see, SIGCSE has an anti-harassment policy (which is great) but I can’t find anything about guns. I think that far too little heed is paid to the intimidatory nature of someone with a visible gun starting a discussion/argument with a speaker. If we are prepared to stop speakers being stalked, why are we prepared, as an educational community, to allow them to be intimidated by visible firearms?

I am, genuinely, considering whether I should be attending conferences in the US in places where the gun control laws are so at odds with what I’m used to at home. I have a lot to think about on this one and I wonder if this has been brought up with the convenors? Should the international community be thinking more about this as an ongoing issue with attending conferences in the US?

(Please, if I’ve got any of the facts wrong – leap in below and I’ll fix them. Note that “Oh, but KC is perfectly safe” is not actually a fact as KC is in the top 25 most-dangerous cities for gun violence in the US.)


Today, I am furious.

[Not tagged as education, for once. This is still part of my philosophy but has no direct teaching relationship.]

It turns out that my mention of Anita Sarkeesian (@femfreq) in a previous blog was horribly prescient. It now turns out, after having death threats made against her (again) for a talk she was going to give in Utah, that Utah’s concealed carry laws meant that any number of people in the auditorium could have been legally carrying weapons and the police could not scan for these or remove them as it was public space. As a result, she has cancelled her talk because she is, quite understandably, wanting to stay alive.

Most of you will know the amount of trolling that the owner of this site has endured for (perfectly reasonably) pointing out that women are badly represented and served by most video games.

Most of you will know the amount of trolling that the owner of this site has endured for (perfectly reasonably) pointing out that women are badly represented and served by most video games.

I am incandescently angry today. Because, once again, for having the audacity to say that “Video games have some pretty stupid female stereotypes” and advocating female equality, Sarkeesian has once again been threatened in a vile and cowardly manner and she has had to take sensible steps to protect her own life. Like the other poor women caught up in the sewer of GamerGate who have had to leave their homes because cowardly attackers have published their home addresses and exhorted people to rape and murder them for, basically, being women.

There are many disingenuous arguments being bandied around under GamerGate but, if you look, you’ll note that they’ve been discredited, so now we’re just down to that game that weak men never seem to get tired of – blaming and attacking women because they feel out of control.

I’m sure that some people will say “Ah, but the presence of good guys with guns in the audience will mean…” and then you will stop because, having thought this through, you will realise that human reaction times and the fact that most gun carriers have not trained for urban conflict means that an attacker can stand up and shoot the speaker before anyone does squat. Yes, great, then all of the handgun heroes can shoot down the bad guy – Pow! Pow! Except that people aren’t fungible and gun vengeance will not miraculously bring the speaker back to life. In Utah, guns were used in 51% of all murders in 2011, an 18% increase over 2010, so people have a right to be scared of guns when a death threat (for a ‘massacre’) has been issued. As Ms Sarkeesian wrote herself on Twitter:

To be clear: I didn’t cancel my USU talk because of terrorist threats, I canceled because I didn’t feel the security measures were adequate.

The only way to stop guns being used in a venue is to remove the guns. When a death threat has been issued, the easy availability of guns makes them the most likely form of effective implementation of that threat. But, apparently, Ms Sarkeesian did not deserve a safe venue and her talk had to be cancelled.

(Before anyone starts on my gun fears, I was a soldier and have fired everything from 9mm pistol up to tank weapons, including rocket launchers and grenades. I have no fear of weapons, I just don’t trust many people who carry them around with them all the time for what appears to be no reason, especially in the terribly fragile urban environment.)

This is a terrible day for everyone. As a species, we have failed to protect women. As a state, Utah has failed to provide safety for people wishing to freely express their opinions. As a group, the thugs and bullies who have been harassing women are making the world a horrific place.

A number of you will stop following me today. Ok. A number of you will want to say “Ah, but those women…” to which I say “Shut up and come back to me when you have a sound justification for rape and death threats in the face of criticism, you idiot.” Today I have no patience with dissemblers, “Devil’s advocates” (seriously, he needs help?) and men who want to blame everything wrong in their lives on women who dare to keep striving for an equality that they have not yet achieved. Today I have no time for people who make arguments that more guns make things safer, when all that more guns do is put more guns into the equation, especially in spaces where a known threat is in effect.

I am already worried that, by writing this, I have made myself a target. There are people out there, searching for women and their supporters, so they can bring harassment to our doors, expose our personal details, and drive us off the Internet so that the only voices are theirs. I am proud to be a supporter of women, of their ongoing fight for equality, and for the perfectly legitimate cause of feminism. Tomorrow, I may be regretting this because some little person with a computer has decided to ruin my life. I’m scared and there is no way on Earth that my tiny blog, read by hundreds of people, should have such weight that I should be worried. And, yet, I am because the people who are attacking women and their supporters have made it clear that there is no low point to which they will not stoop in order to silence people that they don’t agree with.

We have built a miraculous machine to send information around the world in microseconds and we are using it to hurl our faeces. If you ever wanted proof of our descent from a common ape ancestor, there it is.

Today, I am furious. And so should you be. A woman was silenced by threats and people stood by and did not do enough to protect her. We should all be furious because this is not the world that we want. Let the last words here be Anita Sarkeesian’s:

I’m safe. I will continue my work. I will continue speaking out. The whole game industry must stand up against the harassment of women.


5 Things: Blogging

I’ve written a lot of words here, over a few years, and I’ve learned some small things about blogging. There are some important things you need to know before you start.

  1. The World is Full of Dead Blogs. There are countless blogs that start with one or two posts and then stop, pretty much forever. This isn’t a huge problem, beyond holding down usernames that other people might want to use later (grr), but it doesn’t help people if they’re trying to actually read your blog sometime in the future. If you blog, then decide you don’t want to blog, consider cleaning up after yourself because it will make it easier for people to find your stuff when you actually want that to happen. If you’re not prepared to answer comments but you still want your blog to stand, switch off comments or put up a note saying that you don’t read comments from here. There’s a world of difference between a static blog and a dead blog. Don’t advertise your blog until you’ve got a routine of some sort going, just so you know if you’re going to do it or not.

  2. Regular Blogging is Hard. It takes effort and planning to pump out posts on a schedule. I managed every day for a year and it damn near killed me. Even if you’re planning once a week/month, make sure that you have a number of posts written up before you start and try to always keep a couple up your sleeve. It is far easier to mix up your feeds to keep your audience connected to you by, say, tweeting small things regularly and writing longer pieces that advertise into your Twitter feed semi-regularly. That way, when people see your name, they realise that you’re still alive and might read what you wrote. If you can, let people know roughly how often you’ll be writing and they can work that into their minds around what you’re writing.
  3. Write What You Want To Write But Try To Be Thematic. It’s easy to get cynical about things like how many people are following you and try to write what other people would like to read. Have some sort of purpose (maybe 2-3 different themes tops) so that what you write feels authentic to you, fits into your interests and is on a small range of topics so that people reading know what to expect. I stretch the rubber band on education a lot but it still mostly fits. I have a different blog for other things, which is far less regularly updated and is for completely different things that I also want to write. Writing somebody else or on something that you don’t really know about almost always stands out. Share your passion in your own way.
  4. Be Ready For Criticism. At some stage, someone is not going to like what you write, unless you are writing stuff that is so lacking in content or that is so well-known that no-one can argue with it. If you express an opinion, someone is probably going to disagree with you and if they do that rudely then it is going to sting. Most people who are reading you, and yes you can see the number of readers, will read you and, if they say nothing, have no strong feelings or probably agree with you to some extent. People are more likely to comment if they have strong disagreement and many of the strong disagreers in the on-line community are card-carrying schmucks. Some of them are genuinely trying to help but there are any number of agendas being pushed where people are committed (or paid) to jump on on-line fora and smash into people holding discussions. Some people are just rude bozos who like making other people feel bad. Hooray.

    SADLY, YOU MUST HAVE A STRATEGY PREPARED FOR THIS. I don’t condone this, I’m working to change it and I think we have a long way to go in our on-line social structures. However, right now, it’s going to happen. Whether this means that you will take steps if people cross lines and become genuinely abusive, or whether you have other strategies, think about what will happen if someone decides to have a go at you because of something you wrote. The principles of freedom of speech start to fall apart when you realise that some people use their freedom to remove that of other people – which is logically nonsensical. If someone is shouting you down in your own space, they are not respecting your freedom of speech, they are not listening to you and they are trying to win through bullying. John Stuart Mill would leap up from the grave and kick them in the face because his idea of Freedom of Speech was very generous but was based on a notion of airing bad ideas in order to replace them with good. If he had been exposed to the Internet, I suspect he would have been a gibbering wreck in two days.

    Most of you will know the amount of trolling that the owner of this site has endured for (perfectly reasonably) pointing out that women are badly represented and served by most video games.

    Most of you will know the amount of trolling that the owner of this site has endured for (perfectly reasonably) pointing out that women are badly represented and served by most video games.

    Remember: someone else who feels strongly can always start their own blog to air their views. You do not owe idiots space on your comments just so they can abuse people who agree with you or spout nonsense when they have no intention at all of changing their own minds. You going mad trying to be fair is completely unreasonable when this is the aim of the Internet Troll.

  5. Keep It Short and Use Pictures. This is the rule I have the most trouble with. I now try to limit myself to 1,000 words but this is, really, far too long. Twitter works because it can be scanned at speed. FB works for longer things that you are bringing in from elsewhere but falls apart at the long form. However, long blogs get ranty quickly and you are probably making the same point more than once. Pick a size and try to stick to it so your readers will know roughly what they are committing to. Pictures are also easy to look at and I like them because they throw in humour and colour, which break up the words.

    Don't just do image searches randomly - "digital image short" returns this highly rewarding item.

    Don’t just do image searches randomly – “digital image short” returns this highly rewarding item. Trust me when I say that you shouldn’t just image search for “short”. Trust me.

There’s a lot more to say but I’m more than out of words! Hope this helped.

 


5 Things: Ethics, Morality and Truth

Sometimes the only exposure my students will have to the study of ethics is (sorry, ethical philosophers) me and my “freeze-dried, snap-frozen, instant peas” version of the study of ethical issues. (In the land of the unethical, the mono-principled man is king?)

Tasty, tasty, frozen peas. Hey, is that Diogenes?

Tasty, tasty, frozen peas. Hey, is that Diogenes?

Here are a quick five things that loosely summarise my loose summaries.

  1. Ethics, Morals and Truth are Different Things. Morals are a person’s standards of belief concerning acceptable behaviour (we often throw around words like good and bad here). Ethics are the set of moral principles that guide a person’s behaviour or that of a group. Truth is the set of things that are real and factual, or those things that are accepted as true. Does that clear it up? Things that are true can be part of an unethical set of beliefs put together by immoral people. Immoral people can actually behave ethically consistently while still appear unethical and immoral from your group. Ethics often require you to start juggling things to work out a best or most consistent course of action, which is a luxury that we generally don’t have with the truth.
  2. Being Good is Not the Same Thing as Trying to Do the Right Thing. Trying to do the right thing is the field where your actions are guided by your ethical principles. Trying to be the best person you can be (Hello, Captain America) is virtue ethics. Both being good and doing the right thing can be guided by rules or by looking at outcomes but one is concerned who you are trying to be and the other is concerned with what you are trying to do. Yes, this means you can be a total ratbag as long as you behave the right way in the face of every ethical dilemma. (My apologies to any rats with bags.)
  3. You Can Follow Rules Or You Can Aim For The Best Outcome (Or Do Both, Actually). There are two basic breakdowns I’ve mentioned before: one follows rules and by doing that then the outcome doesn’t matter, the other tries to get the best outcome and this excuses any rules you break on the way to your good outcome. Or you can mix them together and hybridise it, even throwing in virtue ethics, which is what we tend to do because very few of us are moral philosophers and most of us are human beings. :)
  4. Consistency is Important. If you make decisions one way when it’s you and another way when it’s someone else then there’s a very good chance that you’re not applying a consistent ethical framework, you’re rationalising. (Often referred to as special pleading because you are special and different.) If you treat one group of people one way, and another completely differently, then I think you can guess that your ethics are too heavily biassed to actually be considered consistent – or all that ethical.
  5. Questioning Your Existing Frameworks Can Be Very Important. The chances that you managed to get everything right as you moved into adulthood is, really, surprisingly low, especially as most ethical and moral thinking is done in response to situations in your life. However, it’s important to think about how you can change your thinking in a way that forms a sound and consistent basis to build your ethical thinking upon. This can be very, very challenging, especially when the situation you’re involved in is particular painful or terrifying.

And that’s it. A rapid, shallow run through a deeply complex and rewarding area that everyone should delve into at some stage in their lives.


Tukhta: the tyranny of inflated performance figures.

I’m sketching out a book on the early Soviet Union and artistic movements (don’t ask) so I’ve been rereading every Russian author I can get my hands on. I read a lot of these works when I was (probably too) young, starting from the very easy and shallow slopes of “Ivan Denisovich” and then plunging down into “Gulag Archipelago”. One of the things that comes out starkly from Solzhenitsyn’s account of the forced labour camps of “Gulag Archipelago” is the way that unrealistic expectations from an overbearing superior organisation can easily lead to an artificial conformity to productivity requirements, which leads to people cheating to achieve their overly ambitious quotas. In Solzhenitsyn’s words, the many thieves in the camp (he is less than complementary about non-political prisoners) coined the word tufta, which he rendered into better Russian as tukhta, the practice of making up your quotas through devious means and fabricating outputs. This could be as simple as writing down a figure that didn’t reflect your actual labour or picking up a pile of timber that had already been counted, moving it somewhere else, and counting it again.

The biggest problem with achieving a unreasonable goal, especially one which is defined by ideology rather than reality, is that it is easy for those who can to raise the expectation because, if you can achieve that goal, then no doubt you can achieve this one. This led to such excesses as the Stakhanovite movement, where patently impossible levels of human endeavour were achieved as evidence of commitment to Stalinist ideology and being a good member of the state. The darker side to all this, and this will be a word very familiar to those used to Soviet history, is that anyone who doesn’t attain such lofty goals or doesn’t sign up to be a noble Stakhanovite is labelled as a wrecker. Wreckers were a very common obstacle in the early development of the new Soviet state, pointing out things like “you can’t build that without concrete” or “water flows downhill”.  It should be noted that the original directives of the movement were quite noble, as represented in this extract from a conference in 1935:

The Stakhanovite movement means organizing labor in a new fashion, rationalizing technologic processes, correct division of labor, liberating qualified workers from secondary spadework, improving work place, providing rapid growth for labor productivity and securing significant increase of workers’ salaries.

Pretty good, right? Now consider that the namer of this movement was “Aleksei Stakhanov, who had mined 102 tons of coal in less than 6 hours (14 times his quota)”. This astounding feat of human endeavour was broken a year later, when Nikita Izotov mined 607 tons of coal in a single shift! It’s worth noting that fully-mechanised and highly industrialised contemporary Australian coal mines can produce round about 3,800 tonnes every 6 hours. What a paltry achievement when all you need is six Nikita Izotovs. So this seemingly well-focused initiative, structured as a benefit to state and worker, is disingenuous for the state and dangerous for the worker.

""Stakhanovite model soviet worker guarantees the continuing peace!"" You'll note the anti-intelligensia and racist targets of the worker - ideologically these were all wreckers.

“”Stakhanovite model soviet worker guarantees the continuing peace!””
You’ll note the anti-intelligensia and racist imagery on the poster as well – ideologically these were all wreckers.

Imagine that you are a worker trying to keep yourself and your family alive in the middle of famine after famine – of course you want to meet the requirements as well as you can, potentially even exceeding them so that you don’t get sent to a camp, locked up, or demoted and diminished in your role. While some people might be practising tukhta out of laziness, you are practising it because it is the way that things are. You need to nod in agreement with ridiculous requirements and then write up your results in a way that exceeds them, if you want to survive. Your reward? Even more ridiculous requirements, not determined in capacity and available inputs but in required output. Tukhta is your curse and your only means of survival. Unsurprisingly, the Stakhanovite movement was denounced as part of Stalinism later on in the emerging and mutating Soviet Union.

Now imagine that you are a student. You have been given a pile of reading to do, a large collection of assignments across a variety of subjects that are not really linked to each other, and you are told that you need to do all of this to succeed. Are you going to deeply apply yourself to everything, to form your own conceptual framework and illuminate it through careful study? Well, perhaps you would, except that you have quotas to achieve and deadlines to meet and, around you, other students are doing better, pressing further and are being actively rewarded and encouraged for it. Will you be at least tempted to move things around to achieve your quota? Will you prioritise some labour over another, which could be more useful in the long-term? Will you hide your questions in the hope of being able to be seen to not be a bad student?

Now imagine that you are a young academic, perhaps one with a young family, and you are going to enter the job market. You know that your publications, research funding and overall contributions will be compared to other stand-outs in the field, to overall averages and to defined requirements for the institution. Will you sit and mull contemplatively over an important point of science or will you crank out yet another journal at a prestigious, but not overly useful, target venue, working into the night and across the weekend? Will you look at the exalted “Research Stars” who have very high publication and citation rates and who attract salary loadings up to a level that could pay for 2-3 times the number of positions they hold? Will you be compared to these people and found wanting? Will you write papers with anyone prestigious? Will you do what you need to do to move from promising to reliable to a leader in the field regardless of whether it’s actually something you should be doing? (Do you secretly wonder whether you can even get there from where you started and lie awake at night thinking about it?)

Measurements that pit us against almost impossible standards and stars so high that we probably cannot reach them grind down the souls of the majority of the population and lead them into the dark pathways of tukhta. It is easy to say “Don’t cheat” or “Don’t work all weekend” when you are on top of the pile. As the workers in the Gulag and many Soviet Citizens found out, doing that just lets the people setting the quotas to keep setting them as they wish, with no concern for the people who are grist to the mill.

Tukhta should not be part of an educational system and we should be very wary of the creeping mensuration of the academy. You don’t have to look far to see highly celebrated academics and researchers who were detected in their cheating and were punished hard. Yet a part of me knows that the averages are set as much by the tukhtaviks that we have not yet detected and, given how comparative was have made our systems, that is monstrously unfair.

Assessing how well someone is performing needs to move beyond systems that are so pitifully easy to game and so terribly awful to their victims when they are so gamed.


The Part and the Whole

I like words a lot but I also love words that introduce me to whole new ways of thinking. I remember first learning the word synecdoche (most usually pronounced si-NEK-de-kee), where you used the word for part of something to refer to that something as a whole (or the other way around). Calling a car ‘wheels’ or champagne ‘bubbles’ are good examples of this. It’s generally interesting which parts people pick for synecdoche, because it emphasises what is important about something. Cars have many parts but we refer to it in parts as wheelsI and motor. I could bore you to tears with the components of champagne but we talk about the bubbles. In these cases, placing emphasis upon one part does not diminish the physical necessity of the remaining components in the object but it does tell us what the defining aspect of each of them is often considered to be.

Bubbles!

Bubbles!

There are many ways to extract a defining characteristic and, rather than selecting an individual aspect for relatively simple structures (and it is terrifying that a car is simple in this discussion), we use descriptive statistics to allow us to summarise large volumes of data to produce measures such as meanvariance and other useful things. In this case, the characteristic we obtain is not actually part of the data that we’re looking at. This is no longer synecdoche, this is statistics, and while we can use these measures to arrive at an understanding (and potentially move to the amazing world of inferential statistics), we run the risk of talking about groups and their measurements as if the measurements had as much importance as the members of the group.

I have been looking a lot at learning analytics recently and George Siemens makes a very useful distinction between learning analytics, academic analytics and data mining. When we analyse the various data and measures that come out of learning, we want to use this to inform human decision making to improve the learning environment, the quality of teaching and the student experience. When we look at the performance of the academy, we worry about things like overall pass rates, recruitment from external bodies and where our students go on to in their careers. Again, however, this is to assist humans in making better decisions. Finally, and not pejoratively but distinctly, data mining delves deep into everything that we have collected, looking for useful correlations that may or may not translate into human decision making. By separating our analysis of the teaching environment from our analysis of the academic support environment, we can focus on the key aspects in the specific area rather than doing strange things that try to drag change across two disparate areas.

When we start analysis, we start to see a lot of numbers: acceptable failure rates, predicted pass rates, retention figures, ATARs, GPAs. The reason that I talk about data analytics as a guide to human decision making is that the human factor reminds us to focus on the students who are part of the figures. It’s no secret that I’m opposed to curve grading because it uses a clear statement of numbers (70% of students will pass) to hide the fact that a group of real students could fail because they didm’ perform at the same level as their peers in the same class. I know more than enough about the ways that a student’s performance can be negatively affected by upbringing and prior education to know that this is not just weak sauce, but a poisonous and vicious broth to be serving to students under the guide of education.

I can completely understand that some employers want to employ people who are able to assimilate information quickly and put it into practice. However, let’s be honest, an ability to excel at University is not necessarily an indication of that. They might coincide, certainly, but it’s no guarantee. When I applied for Officer Training in the Army, they presented me with a speed and accuracy test, as part of the battery of psychological tests, to see if I could do decision making accurately at speed while under no more stress than sitting in a little room being tested. Later on, I was tested during field training, over and over again, to see what would happen. The reason? The Army knows that the skills they need in certain areas need specific testing.

Do you want detailed knowledge? Well, the numbers conspire again to undermine you because a focus on numerical grade measures to arrive at a single characteristic value for a student’s performance (GPA) makes students focus on getting high marks rather than learning. The GPA is not the same as the wheels of the car – it has no relationship to the applicable ability of the student to arbitrary tasks nor, if I may wax poetic, does it give you a sense of the soul of the student.

We have some very exciting tools at our disposal and, with careful thought and the right attitude, there is no doubt that analytics will become a valuable way to develop learning environments, improve our academies and find new ways to do things well. But we have to remember that these aggregate measures are not people, that “10% of students” represented real, living human beings who need to be counted, and that we have a long way to go before have an analytical approach that has a fraction of the strength of synecdoche.


Follow

Get every new post delivered to your Inbox.

Join 804 other followers