From read only, to read/write, to write only, to … ?

Now this is long, if you want the 5c version, software developers using Generative AI are starting to generate large amounts of automated text that takes a long time to read, which means that the people checking their software are using GenAI to summarise it and accept the changes. This means that the human role in software oversight is potentially reducing, and quickly. Now to the long version.

The long-winded intro

The interactivity of networked communication has been an amazing development in human information transfer. Books, which are great, are large, physical objects, where you can read what I wrote but you can’t (easily) respond to me in a way that other people can see your thoughts on my writing. It was slow to write books with ink and pen and even the advent of the typewriter and word processor has only sped up the creative act of producing the text of a book by around a factor of five. Printing a book has got much faster, certainly, and electronic distribution of the written word has made it almost instantaneous from decision to print to “here is the book”, but it still takes time to write a book. If you assume a book length of 80,000 words, it’s about 133 hours of handwriting to to produce those words, or about 27 hours if you’re not a bad typist and don’t stop to eat, sleep, or breathe. This doesn’t allow any time for thinking, plotting, drawing complex diagrams, or edits, but it’s a way to think about how much human time and effort is required to write a book. I once wrote 50,000 words in just over 20 hours for a charity writeathon and I can tell you now that I could barely talk when it was done. Many authors write somewhere around 1,000 words a day, which would mean you’re waiting close to three months for a book. Actual writing, with content and thinking, takes time.

On the receiving end, humans read around 260 words per minute, far faster than people type and anyone handwrites, with that 80,000 word book taking a little over 5 hours to read (you have a comfortable chair). You don’t have to write a response to the book, you just read it, think about it, and you’re done. Of course, we’ve been sending each other much shorter messages than books for ages – messages sent by messenger, courier, or postal service to exchange business information and personal thoughts, have been around for thousands of years. But both of these have one thing in common in that the written entity is given to us and we never really modify it. A letter exchange has input from both sides but through the exchange of individual read only elements, with the time between letters determined by how long the first took to write, that one took to send, the time for the receiver to read and craft a response, and then choose to send it again. Individual letters could be collected, physically, or by being printed into a volume, but they remained a set of independent read-only messages.

One of the first new communication models to emerge after we started connecting computers together was simple messaging, then e-mail and discussion boards, and it’s in discussion boards that we started to see a read/write aspect to a conversation really emerge. Discussion boards provided persistent containers for the organisation of messages and responses to those messages (and responses to responses…), suddenly changing an information exchange between multiple people into something that could be changed by both parties. The ability to edit and change shared pages took this even further – Wikipedia, for example, is something many people read, but what you read is the product of hundreds or thousands of people in an ongoing conversation as they change any or all parts of the information on the page. LiveJournal, a long form social media site from the past, was often joked about as being “write only”, because users would produce long articles about things that mattered to them, shared onto a feed where others could read (and comment), but they ignored any comments that came their way as they weren’t interested in dialogue but broadcast. Books are broadcast. A set of letters can be a conversation, if stored correctly. A discussion board or social media site provides mechanisms to support a useful dialogue, with varying levels of editing older information, although sometimes it’s just people shouting at clouds.

Oh, Bruv New World

Under all of this is a simple assumption, that all the words were created by someone, with an intention of being read by someone else. This assumption broke a few years ago and this breakage is having some serious ramifications across a range of spheres.

The advent of generative AI means that novel length words can be generated in the blink of an eye by a short prompt, channeling the stolen work of millions of people to pump out a notional novel in seconds. It no longer takes 27 hours of typing, and goodness knows how many days of thinking, to produce a novel. Of course, another aspect of generative AI is using it to summarise works or reports for you, which means that it no longer requires 5 hours to read a novel – you can get a summary in seconds. This, of course, would have an immediate effect on fiction publishing, especially if people are happy to read AI generated work. If someone is still reading manually, there are only so many books that they can read in a year – which means the more AI generated ones they read, the fewer human ones they can read.

I don’t know quite what to call a model where an AI writes the book, and a human claims authorship, and then another person uses AI to summarise the book, and says that they’ve read it. It reminds me of the human/robot discussions of the late science fiction writer Philip K. Dick, who wrote about the blurred lines between creatures that say that they’re one thing but may be another. For me, and no doubt many will disagree, authorship requires both the creative intent and the work to bring it to fruition – ideas are cheap, effort is what brings them to the world. Someone writing large books using a small AI prompt is not, in my mind, an author. But I don’t have another word for it quite yet. I do feel that someone who is only reading short summaries of books has not actually read the book – they may have the gist, or even a good idea of the ending, but they do not have the knowledge of the way that the book was written or any idea of the many themes and nuances that were used in the book. Sure, they’re still reading but have they read this book? Sorry, I don’t think so.

Is this really new?

The generative contribution to verbosity is new. From the moment we made it easier for people to add volume to their text through cut and paste, whether physical or electronic, texts have been getting bigger. When legal judgments went from hand written to computerised, pages and pages of former cases could be added at no effort to the crafting judge. While this notionally added important information, this low effort inclusion of additional information put a greater burden on anyone seeking to read the judgment later. Before generative AI, this burden was shouldered by junior lawyers and clerks, as summarising the case required someone human to do the work. After generative AI… it has become far easier to generate a large volume at the start and then summarise it to (notionally) salient points without all the human labour. I say notionally as, of course, LLMs are prone to error and hallucination, and any generation or summary could be wrong. If a human is doing the summary, then they may spot a generative error – may.

At its core, we have to look at the role of the human in here. There is a finite limit to the speed with which humans can read and write, which limits the speed of producing and analysing written texts. No argument. But speeding this up, using non-human means, should probably not be at the loss of accuracy, quality, and usefulness of the information in question. The slowness of humans is a limit, not a benefit, but the price of using human intellect, training, and expertise is that they have a speed limit. We probably don’t want our legal judgments to devolve to “hey, Siri, give me 10 pages of why he’s guilty”, and “yo, read these 10 pages, summarise to one sentence and three bullet points for case for appeal.” The “didn’t bother to write it”/”not going to bother to read it” model has got some pretty obvious pitfalls in it, even in the simplest of cases.

Real World Impact

The didn’t write/didn’t read model is one we can look at in fiction as a curiosity, and get a slightly worrying feeling about it’s use in law, but there’s another place where DW/DR can actually cause trouble and that’s in the area of software development. For those who don’t know about large scale software engineering, it’s often built on a repository model, where the pieces of code are stored in a central repository that tracks versions, allows integrated testing, developers to exchange comments and raise issues, and, ultimately, makes large and intricate code bases easier to look after. A number of these are built on software called git, which is a distributed version control system, and developers can take a piece of code, change it, test it, and then, when it’s accepted, merge it back into the main software so that their changes are used in the current production version.

The online platform GitHub uses Git with a large amount of project management wrapping to let people from all over the world work on code in a controlled and systematic way. Ideally, for a given section of code, any changes are only being made by one person, all changes are properly tested, and then these changes are presented back to a moderator of some sort to check so that one person changes and one other person checks. The developer works on a branch pf the main software, makes changes, is happy that it works, then uploads it back to the server. Then they issue a pull request, where they compare what they’ve done to the main branch, add a description of what they’ve done, tag any issues they believed they’ve fixed, and ask someone (or nominate) someone to review their new code. This code is then run through automated tests and continuous integration systems to make sure it doesn’t break immediately, then what should happen is that a human reads through the PR, checks the code, writes any comments, and, once approved, the code is then merged to the main branch. This code is now part of the main build.

But what has started happening with the use of Generative AI is that the effort required to write long and copious notes on software changes is diminishing. If GenAI is generating your PR, it can lead to longer descriptions and longer issue statements. Now this wouldn’t be a problem by itself, except that the frequency of PRs are increasing as people use GenAI to write code to solve issues. Now the humans doing the review have got both more text to read in an individual PR AND more PRs. So what are they doing? There’s clear evidence that more people are using GenAI to analyse the code in the PRs and get a tick of approval. In some cases, the GenAI is authorised to do the merge if the PR passes muster.

What this means is that what should have been a benefit, increased code output and increased description, has now started to push people out of the checking loop. Code is being generated so fast, much faster than humans could, that it’s exceeding what the reviewers can handle, which means that they’re having to use Generative AI to keep up. And that means that true human oversight of code changes is reducing. And, remember, LLMs sometimes make things up, sometimes are wrong, and are often random in unpredictable ways – all code generated by them needs oversight, just as human written code does.

We’ve already seen an unpleasant dehumanising of lower quality educational systems where overworked teachers use GenAI to generate worksheets that performance-driven students use GenAI to complete and which are then marked by GenAI because the teachers don’t have time. A scenario where not only do we not have confidence that anyone is learning anything, but we literally have no way of knowing because all of it is being done by AI. This is potentially damaging and dangerous to students if left unchecked.

Now imagine a software future where Claude writes the code, ChatGPT writes the PR, Copilot does the checks, ChatGPT writes some comments, Claude notionally responds and then merges this to the main branch. Let’s assume that each of these is 95% accurate, the chances of this code and all of its documentation being correct is around 77%. Now humans are certainly not perfect but they are, in this setting at least, also not random in their solution seeking. Programmers work on a design, where a lot of the work in the software is done before any code gets written, often using tools such as very specific types of diagram, relational modelling, and markup languages, to ensure that the code, when written, is more likely to be accurate. Also, skilled software developers can be very good at spotting bugs in code and there is an intentional mental discipline and a set of semantics that underpin the writing of code – humans do not write code randomly and we know that they can have a far more detailed mental model of a system than any current LLM can manage.

After having six Severity 1 incidents in a week, Amazon backpedalled rapidly on their unsupervised deployment of GenAI-assisted code and now senior engineers have to sign off on everything. Suddenly, the Sev 1 rate dropped. Hmm.

Every time that I read about how much faster we can write code with GenAI, I am reminded of the Simpsons episode where Bart is trying to demonstrate his intelligence by playing chess against three people simultaneously (“The PTA Disbands!”, Season 6, Episode 21). Immediately after someone saying “Hey, that boy is playing three games at once!” he is checkmated on all boards, as he may be playing three people but he’s not playing well. GitHub has fallen over many times for capacity in the last two years as the change rate has exploded with GenAI code changes being pushed, but we’ve also seen a sharp spike in code revisions, where merged changes have had to be removed because they didn’t work and none of the tools used picked that up.

There are ways of using these tools to reduce some grindlike aspects of programming but they are nowhere near accurate enough to operate without some form of human oversight, and this means that humans have to be able to read and digest all of the associated documentation and code changes. The “Didn’t write/didn’t read” model is hard to justify in any setting but it’s extremely dangerous in the area of programming, as it is not mature enough to produce working systems at the same level of reliability as before, and it robs everyone involved of the opportunity to develop and maintain their skills.

I fear that we are just playing bad chess on millions of boards and that there is a looming wall of checkmates.