Grand Challenges in Education – When we say grand, we mean GRAND!

Highly coloured picture of a piano

Some time ago, Mark Guzdial posted on the Grand Challenges in the US National Educational Technology Plan. If I may summarise the four, huge, challenges, they were:

  1. A real-time, self-optimising difficulty-adjusting, interactive learning experience delivery system.
  2. A similarly high-end system for assessment of cross-discipline complex aspects of expertise and competencies.
  3. Integrated capture, aggregation, mining and sharing of content, learning and financial data across all platforms in near real-time.
  4. Identify the most effective principles of online learning systems and on/offline systems that produce equal or better results than conventional instruction in half the time and half the cost.

Wow. That’s one heck of a list. Compare that with the list of grand challenges from the March, 2011, report of National Science Foundation Advisory Committee for Cyberinfrastructure Task Force on Grand Challenges, which defines the grand challenge problems for my discipline, Computer (Cyber) Science and Engineering. By looking at some very complex problems, they arrived at the following list of areas in which great strides can, and should, be made:

  1. Advanced Computational Methods and Algorithms
  2. High Performance Computing
  3. Software Infrastructure
  4. Data and Visualisation
  5. Education, Training and Workforce Development
  6. Grand Challenge Communities.

Let me rewrite this last list in simpler, discipline free, terms:

  1. Better methods for solving hard problems.
  2. Big machines for solving hard problems.
  3. Good systems to run on the big machines, to support the better methods.
  4. Ways to see what results we have – people can see the results to make better decisions.
  5. Training people to make steps 1-4 work.
  6. Bring people together to make 1-5 work better with greater efficiency.

Now, lets look back at the four USNETP educational grand challenges to see if we can as easily form such a cohesive flow – we want to be able to see how it all works together.

  1. Smart learning systems.
  2. Smart assessment systems.
  3. Data and Visualisation. (Nick note: get into data and visualisation! 🙂 )
  4. Fusing the best of the old and the best of the new.

Now, the USNETP focus is on useful R&D and these challenges are part of their overall view of “they all combine to form the ultimate grand challenge problem in education: establishing an integrated, end-to-end real-time system for managing learning outcomes and costs across our entire education system at all levels. ” but what immediately leaps out at me are the steps 5 and 6 from the previous list. Rather than embed the training and community aspects somewhere in the rest of a document, why not embrace this at the same level if we’re talking about grand challenges in Education? That would give us:

  1. Training educators to make steps 1-4 work.
  2. Forming communities of practice to make 1-5 work better with greater efficiency.

Now these last two steps, of course, are what we’re doing with the conferences, the journals, the meetings and blogs like this but it makes a lot of sense when we see it inside my discipline, so it seems to make sense in the general field of education. There’s no doubt that these two last steps are easily as hard to manage at scale as the other projects, even interoperating with them. In fact, by making them huge challenges we increase their worthjustify effort and validate the research community built up around them. These are financially-sensitive times, where academics have to provide a value for their work. Allocating these important tasks to the grand challenge level recognises the difficulty, the uncertainty of being able to solve the problem and the sheer amount of work that may be involved.

These are, of course, only my thoughts and I have a great deal to learn in this space. I’m still searching for answers but if there’s a nice convenient report that says “Well, duh, Nick, we’re doing that right here, right now” I look forward to correction and enlightenment.

But, if it’s not already part of the USNETP grand challenges – what do you think? Should it be?


Teaching Tools (again): Balancing Price, Need and Accessibility.

I’ve spoken before on open source and teaching tools but I’ve been reviewing some interesting data on textbook purchasing. As some of you may know, book purchases are dropping in many areas because students feel that they don’t need to (or can’t afford to) buy the text. Some of the price burden of textbooks is the size, printing and shipping costs associated, so eBooks, which can be and often are, cheaper should be addressing this problem.

Is that our experience? Well, we’re still collecting data but, anecdotally, no. Despite eBooks being substantially cheaper, students aren’t buying them in any greater numbers. (Early indications are that it may actually be less.)

Price is always going to be an issue. 60-70% of a large number is still a large number (to a student).

Need is an issue – do students need the book as a text or a reference? Will they be able to get by on lecture notes? How is the course structured? There are important equity issues associated with forcing a student to buy a book as you don’t know what they’ve had to give up to do that, and the resale market for secondhand books is not what it once was.

But one of the big concerns of my students is accessibility. They are well aware that buying an electronic book may give it to them in a very constrained form – a book that can only be read on one machine and may not survive upgrades, a book that may not have a useful search mechanism, a book where you can’t easily highlight the text. Worse, it may be a book that, sometime in the future, just stops working and can never be read again.

Yes, publishing companies are pouring millions of dollars into solving this problem but books are special in a very important way. Books enable knowledge transfer, they don’t own or restrict the knowledge transfer. When you produce a physical book, people can expend effort to do what they like. Make a house out of it, read it, re-index it, tear out the pages and put them together in print density order. None of this is possible with an eBook unless someone lets you. (Ok, you can build a house but it will use your laptop or tablet.)

I can’t help thinking that most of the effort seems to be going into providing the experience that publishing companies want us to have, in terms of usage, ownership and access – focusing on controlling us rather than enabling us. Perhaps this is the point we should address first?

(If you haven’t read my post on Hal Abelson’s talk, you might want to get to that after this.He talks a lot about the problems with the walled garden and his terminology,including the very useful term generative, is a very interesting read.)


The Facts of Undergraduates

If you’re driving through the city and you’re in a hurry, you’re probably going to arrive at your destination and be under the impression that every light was red. You notice the red lights because they get in your way – worse, you may have convinced yourself that you need greens all the way and anytime that this situation doesn’t occur you have to deal with your expectations being thwarted.

Some of you, reading this, are getting angry or frustrated just reading this.

Setting up false expectations is the best way to get disappointed, frustrated and angry. I always set myself up with a positive mindset before lecturing or student contact, because starting a lecture or a meeting angry or frustrated is just going to set you up so that every negative interaction makes it worse. But, throughout an entire course, there are certain things that some people are going to do, we can’t just prepare for a course full of quiet, always attentive, highly intelligent, well-prepared and engaged students. Let’s look at a short list of other behaviours and my list of positive preparation for it.

  1. Students may not show up to lectures. Some students will stop coming and won’t come back – a lot of students will come back if the lecture is useful, interesting AND you have things like recordings or notes to let people catch up with missed lectures. Unless 100% attendance is required (and the question there is always “Why? Where is the educational value?”), giving people a mechanism to get back in is probably going to work better than holding a hard line. Recordings, and podcasts, let you collect your thoughts and review what you said – these help you as much as they help students.
  2. Students may not prepare. Once you start requiring preparation and you give students some value for that (participation, marked quizzes, things like that) students tend to re-optimise and prepare. This is a great opportunity to add some formative or small summative exercises in that can get great discussion or participation going.
  3. Students aren’t always focused on your lecture. There are many things going on for our students and they’re trying to work out where to spend their effort. This is a challenge, sure, but what a great opportunity to invest enthusiasm, talk to people and try to bring out the passion that brought at least some of your students in. Getting students talking to each other gives you a huge scaling factor and a communications network where your students can’t hide.
  4. Students just don’t do assignments sometime. I’m a great believer in giving a clear indication of what is required at the start of the course. This isn’t just me being prescriptive, this is a fantastic opportunity for me to review my expectations, my thoughts, my grading schemes, review changes, integrate new content. My course profile is not just a way to let students know what will happen if they don’t do things, it lets me frame the whole course to lead students into the necessity of the assignments and integrate that knowledge from lecture to tutorial to assignment to final exam.

Accepting that some things will not match your vision of the ideal student doesn’t mean that we have to walk into every lecture under a thundercloud. Yes, there’s effort involved, but there almost always is to achieve a good outcome.


Thinking About Students: What a Student DOES Rather Than What a Student IS.

Yesterday’s post briefly discussed Alfred Korzybski and, today, I wanted to talk a bit more about some of his ideas, applied as a method for describing students.

“Why do I need a new way to describe students?” you might well ask. After all, we all know that there are good students, bad students, hard-working students, lazy students…

Or do we?

I only have read a relatively small amount of Korzybski but what struck me was his discussion of the verb ‘to be’ and the way that it could be used in a way that confused someone’s actions with their fundamental identity. For example, if I do something foolish, you could say “Nick is a fool” but what you really mean is “I am calling Nick a fool because he did something foolish.” (There is an old, and obscene, joke along these lines that I shall say no more on.) Committing one foolish act no more makes me a fool than the word “Nick” actually reflects the entirety of my identity.

Similarly, let us consider the movie “Green Lantern”. If I say “the movie is bad”, what I am really saying is “I did not enjoy the movie Green Lantern and would not watch it again, even for free on a plane.” The latter is a fact, based on a subjective opinion, but it is clearly identified as such. Anyone else sharing the longer form would clearly be saying that “I heard Green Lantern was not enjoyable to Nick and, given that I have similar tastes, I believe that I would not enjoy it either.”

How does this apply to students?

It’s easy to talk about good, bad, hard-working and lazy students but this often confuses the facts of a student’s actions with the student themselves. How do we characterise a good student? Michael hands up all of his work on time, has never cheated and achieves high marks. Does that make him a good student? These are all characteristics of a good student, certainly, but by listing these actions in full I make it clear how other students can achieve this aim. If I tell someone to be a good student – or to work harder (to be hard-working effectively) – I don’t actually tell them what to so, I ask them to match an identity or fulfil a predicate, rather than clearly showing them what I expect them to achieve. Of course, there is no guarantee that Michael is a ‘good’ student – but by listing our perceptions of his actions we explain why we might apply such a categorisation.

Referring to a student with ‘This student is…’ risks hiding a factual statement inside a statement that appears much stronger and has much wider impact, but without qualification. Now this goes beyond identity and simple statements and extends to the way that we interact with students as well.

“The student is responsible for handing up their work on time” – sounds good, but just saying it is an assertion of what you want to occur. What I mean when I think this is “We expect students to read the deadlines for submission, allow enough time to complete the assignments and submit their work on time, to the correct locations, in the correct format. If you don’t do this, then you will lose some or all of the marks for the assignment and may not be allowed to continue with the course.”

This is, by definition, a discussion of semantics and it is a little bit of me thinking aloud. As a mental exercise, I find it very useful because whenever I want to classify a student as ‘this student is…’ I force myself to think about what the student has done to make me think that way. Quite often, in review, I find terms like ‘bad, lazy, hard-working, good, indifferent, difficult’ dropping away because such simple classifications are beyond me – although not always. You’ll note that a recent post of mine dealt with the ‘rude student’, although I went on to describe why this may or may not be a fair description. Sometimes, for brevity or ease of reference, we may use this form to describe a group or a type, especially where people know what we mean. The problem arises when we make absolute statements about someone from smaller, and occasionally ephemeral, information. Extrapolating to a strong statement when we do not have enough information to do so. And some of these labels will stick – and stick hard – throughout a student’s life.

I suppose that the benefit for me lies in considering everything, good and bad, that a given student has done because it makes me regard them as a person, rather than a simple “is a…” – a being who has taken a number of actions, and may take any of the other possibilities in the future. Somebody that DOES rather than somebody that just IS. Someone with a great deal of unrealised potential and untapped energy. Someone who could do anything rather than being stuck in a box from the misapplication of a strong label somewhere back in their past.

 


The Map is not the Territory.

I’m writing a paper on visualising Internet network topologies with my PhD student and some colleagues at the moment and an old friend, who is one of the student’s other supervisors, looked at some of the work we’d been doing and mentioned a great quote from Alfred Korzybski in 1931:

“The map is not the territory.”

Korzybski was a philosopher and scientist who developed the theory of general semantics, which I’m not going to talk more about here, but a lot of his work revolved around the idea that all we have access to is perceptions and beliefs, which we confuse with a knowledge of actual reality. This is a simple quote and a powerful concept: one of my favourite combinations.

What brought me to this was that, as part of our paper, we were looking at the London Underground map – the famous Tube Map.

A picture of the map of the London Underground

The focus of the Tube map is getting around London by Tube. Designed in 1931 by Harry Beck, a draughtsman with experience in laying out electrical circuits, it replaced a large number of incomplete and more geographically focused maps. What is most interesting about this map is that some licence is taken with the geography in order to make this the simplest map to use for Tube travel. Above ground, this map is not only not as useful, in some areas it’s completely wrong. (Suburbs on the opposite side shown, distances completely inaccurately represented for ease of reading.)

This has had an effect on the way that people travel around London – making decisions above ground that make sense on the Tube map but are downright silly when on foot on the streets. To combat this, Transport for London have developed the Legible London project with above-ground signage to assist the navigation of London Above, with signs and images showing you directions and landmarks.

Whether it’s maps of networks, maps of London or course pre-requisite diagrams, maps are only useful if you design them correctly for their primary use. Looking at the work on prerequisites that I’ve been talking about recently, it’s becoming more apparent that my desire for a good visualisation of pathways stems from my desire for a map that correctly reflects what we want students to do, reinforces the correct behaviour and is also going to be fit for purpose. Rather than using one diagram for many things, I need to check to make sure if I have the best diagram for a given situation.

Sometimes I need to release my grip on the accuracy of geography (precise location) to focus on the detail of topology (arrangement and connectivity). Sometimes it’s the other way around. Particularly when I insert a temporal aspect, I need to make sure that this “fourth dimension” doesn’t make my maps so complex that they’re useless. However, I always need a reason to relax a requirement: I’m certainly not saying that you can scribble randomly on a piece of paper and call it the NYC Subway map!

But, taking this concept further, how many pieces of work are out there that confuse a good diagram or a flowchart with the real thing? Is this just our confirmation of our perceptions and, as as result, it’s strongly sensible only when viewed from within our context? Or are we producing transferrable and shareable maps, focused on the right detail, showing the correct view of the terrain for the purpose, and accepting that there are an almost infinite set of views of the true territory?

A good map helps us to navigate territory but it can never replace it. What I always need to remember is that if I produce a map from a map, I can add no more detail than was in the original and I cannot correct mistakes in the original, without reference to the territory itself. And that’s something I think that is always worth remembering.


Beautiful, Interesting and Good: Information Storage for a New Age

I have just received some information from the Museum of the City of New York. I had been searching for some information on Truman Capote in 1966 and, after being unsuccessful in my original web search, located a reference to it in one of the blogs of the researchers of the Museum of the City of New York. I sent a query about the possible availability of what I was looking for, received more information on who to contact and, just after midnight on the 24th of March, I received not only what I was after but also the accompanying information that was the obvious companion piece. Of course, not knowing until I received one that I needed the other, this saved me time but, as well, left me slightly awestruck that I am roughly 24 hours away from almost the entire visual, social and literary history of New York. I have, at the other end of my e-mail, someone who will not only give me answers but fill in the holes in my knowledge to give me the answers to questions that I should have asked!

This is, quite frankly, amazing. The sheer amount of work, technology, indexing, curation, information management and money that has gone into making this happen is slightly terrifying – yet, here we are.

Ten years ago, I might have been able to find out the name of the librarian, who probably would have had e-mail but perhaps not the knowledge or the ability to send me an upload link to a temporary file holding server. I probably would have had to use a fax to request information, to provide follow-up to e-mail and assert which organisation was my umbrella, and then negotiate access to file transfer services to get the files across. I don’t think that I would have been able to read a blog, follow-up with a post, get the right e-mail by automated reply and then, in less than 18 hours, be downloading from a cloud-based share site!

Twenty years ago, it may have been a personal visit or a fax – and, even then, we may have had to be part of an ongoing formal or financial arrangement for me to waste the time of remote staff searching the stacks for a particular card in a given box. (Roughly 11 years ago, I couldn’t even look inside the NY Public Library without a library card and had to content myself with looking at lions.)

Thirty years ago, this would have been just too hard.

(Yes, the decade boundaries are a little fuzzy and I am aware that Library Science and Information Science have been doing a lot that most people are unaware of, yet perception is important here and not knowing whether something is available, or not knowing how to get to it, are almost as bad as it not being there or available in the first place.)

The fact that this is now available every day doesn’t make it any less amazing – it just means that every day we stay in this technologically advanced place, where beautiful, interesting and good things are stored, index, curated and watched over by a combination of people and… of course, machines of loving grace… is amazing.

When I talk to my students about what it is we do as Computer Scientists, I talk about handling scale, solving problems, developing algorithms and doing amazing things. My students have grown up with things like this and, if I don’t point out some of the amazing things that today just happen, then they miss out on some of the rich heritage of computing – the world before the Internet, the frontier of the early Internet, before the Web, before the Commercial Web, before mobile computing, before you could wonder something in Australia and have an answer from America with no special training or knowledge beyond how to type and send an e-mail.

I don’t focus on good old days – I have to talk and live in the amazing now, training students for the unimaginable future. Today, I have another example of something that is amazing now, and I look forward to an opportunity to teach about it, sometime soon.


I Ran Out Of Time! (Why Are Software Estimates So Bad?)

I read an interesting question on Quora regarding task estimation. The question, “Engineering Management: Why are software development task estimations regularly off by a factor of 2-3?“, had an answer wiki attached to it and then some quite long and specific answers. There is a lot there for anyone who works with students, and I’ll summarise some of them here that I like to talk about with my students:

  • The idea that if we plan hard enough, we can control everything: Planning gives us the illusion of control in many regards. If we are unrealistic in assessment of our own abilities or we don’t account for the unexpected, then we are almost bound for failure. Making big plans doesn’t work if we never come up with concrete milestones, allocate resources that we have and do something other than plan.
  • Poor discovery work: If you don’t look into what is actually required then you can’t achieve it. Doing any kind of task assessment without working out what you’re being asked to do, how long you have, what else you have to do and how you will know when you’re done is wasted effort.
  • Failure to assess previous projects: Learn from your successes and your failures! How much time did you allocate last time? Was it enough? No? ADD MORE TIME! How closely related are the two projects – if one is a subset of another what does this say for the time involved? Can you re-use elements from the previous project? Be critical of your previous work. What did you learn? What could you improve? What can you re-use? What do you need to never do again?
  • Big hands, little maps: There’s a great answer on the linked web page of drawing a broad line on Google maps at a high-level view and estimating the walking time for a trip. The devil is in the details! If you wave your hands in a broad way across a map it makes the task look simple. You need to get down to the appropriate level to make a good estimate – too far down, you get caught up in minutiae, too far up, you get a false impression of plain sailing.

I found it to be an interesting question with lots of informative answers and a delightful thought experiment of walking the California coast. I hope you like it too!


How Do I Model The Students Who Leave?

This is a quick note on one of the problems I face in trying to analyse student data: dealing with students who are only in the system so briefly that I can’t capture much data on them. In my other educational research work I can look at student behaviour in terms of final grades and on-time assignment submission but, in order to try and see the impact of what we’re doing on behaviour, I really have to be able to capture data before and after a change. I then have to try and eliminate all other factors to find a correlation that looks like it’s significant.

In yesterday’s post, I didn’t mention that one of the issues that the Baldwin-Wallace researchers noted was trying to deal with students who gave some initial data and then left the system – how do you incorporate these students in a way that allows you to infer behaviour without introducing the spectre of bias because you’ve inserted dummy data into your system. They had discussed adding another grade type, W or PW, that would allow them to keep students in their data who had left the program early – can you spot the situation that will lead to people leaving early and can we predict the withdrawal from the course based on earlier performance?

I face the same problem in a lot of my assignment submission data. I have 17,000 students in the initial dataset but, after cleaning and removing students who withdraw, that shrinks a lot. Regrettably, this also removes the students that I really want to work with – those who have withdrawn. We use a binary notation as an overview for on-time and late submission, so extending the sequence is straight-forward, but any time we extend the sequence we have to justify it very, very well to make sure that we haven’t introduced too much noise or bias.

There are a lot of good existing techniques and, of course, Bayesian analysis is once again our friend in many ways but I’m now looking at machine learning to provide a very simple two-component partitioning – can I learn to predict who will be in the incomplete group and who won’t? I have to do something about the ‘length’ of the submission history or the most obvious thing the machine will probably learn is that ‘short history == fail’. I’m looking forward to getting onto this research in the very near future, especially if it ca give me insight into those students who are only with us for a short time. I really need a tool and a model that will work within the first 2-3 weeks – it’s a challenge but a fun one.


What Do I Study? What Do I Do? Showing the Path

One of the things I’ve learned from flying a lot is that it’s never as easy to get from one point to another as you think. There are regional hubs, legal connections, affiliations and the many intricacies of which routes are allowed into which countries. There’s a reason that you can either retain the services of a travel agent for a fee or spend a lot of your own time trying to work out the best way to get from A to B. It would be nice if you could fit everything onto one simple diagram and see the best way to go but, even without the commercial concerns, it’s a very hard problem to solve if you’re worried about efficiency rather than connectivity.

We allow our students a lot of latitude in picking their path through their degrees. Although we offer programs that have a core of prerequisites, there are many opportunities for electives – courses that they can pick and choose from. But, on many occasions, students look at the total number of points they require, and the year level, and pick based on interest or short-term goals, rather than any form of long-term vision.

Going back to our airline model, it’s like trying to get to to New York from Sydney by picking the cheapest flight that goes east. Thinking in one-step-ahead terms prevents you from realising the benefits of flights into longer-range hubs, special deals and the round-the-world flight. Technically, optimising your solution so that your next step is the ‘best’ from those available is a greedy algorithm – each step will be optimal but it’s not guaranteed to give you the best overall solution, just a solution.

What would be great is if we could present students with a simple flight path, a map, a poster or an interactive tool that allows them to see where they want to gowhere they’re starting from, and how they could get there based on our courses. I’ve started sketching out some ideas based on this but complexity is proving to be a problem – as expected. I have some sketches of solutions and, when I have something that might be useful, I’ll share it here.


Participation: The Price of Success

In my various roles I have to look at interesting areas like on-line learning and teaching delivery. One of the classic problems in this area is the success of the initiative to get educators and students alike to use the technology – at which point it melts because the level of participation rises so high that the finite underlying resources are exhausted. The resource was never designed as if everyone would want to, and then actually go on to, use it.

I have had someone, seriously, say to me that an on-line learning system would work much better if the students didn’t all try to use it at once.

The same problem, of course, occurs with educators. If no-one is participating in class then you’re pushing a giant rock up hill to make things happen and it’s more likely than not that a lot of what you’re saying and doing isn’t being taken in. If everyone is participating in class, you’ve jumped into the Ringmaster’s hat, you’re constantly fielding e-mails, forum messages and appointment requests. And, of course, at the end of the long day it’s easy to fall into that highly questionable, but periodically expressed, mode of thinking “universities would be great if there were fewer students around“.

A picture of bread and butter.

Students are our bread and butter. (Not literally!)

I’ve attached a picture of bread and butter to drive this point home. Students are what makes the University. Their participation, their enthusiasm, their attendance, their passion, their ennui, the good and the bad things they do. If the systems we build don’t work with our students, or the volume of students, or automatically excludes a group of students because we can any provide resources for 70% of them, then I think that we’ve got something wrong.

Having said that, I get ‘tight budgets’, I understand ‘district funding shortfall’ and I certainly sympathise with ‘very high workloads’. I’m not saying that people are giving up or doing the wrong thing in the face of all these factors, I’m talking about the understanding I’ve come to that the measure of my success as an educator is almost always linked to how much students want to talk to me about constructing knowledge, rather than than just doing assignment work.

It’s one of those things that, if I prepare for, makes my life easier and I can then view that work blip as a positive indicator, rather than go down the curmudgeonly professorial path of resenting the intrusion on my time. Let’s face it, attitude management is as (if not more) important for the lecturers in the class as it is for the students. You want to feel like you’re doing something useful, you’d like some positive feedback and you want to think that you’re making a difference. Framing increased participation as desirable and something that you plan for has certainly helped me manage the increased workload associated with it – because I take is a sign that my effort is paying off.