How Not to Prepare for Reading Tests

Can you prepare for an ‘unseen’ reading test – a test in which you will have to read something you’ve never encountered before, and answer questions on it? To answer this question, we need to consider what is required to succeed in such a test.

Reading consists of two key aspects: your ability to decode – to turn the squiggles on the paper into the sound of language – and your knowledge and vocabulary. Clearly, decoding is crucial, otherwise a reading test will be the equivalent of being asked to read ancient Egyptian hieroglyphics. Beyond that, you need knowledge and vocabulary. Your ability to understand a text depends upon your knowledge of the domain with which the text deals. Most of the readers of this blog probably consider themselves to be good readers, who would perform well in a reading test, but they would struggle to make sense of the following passage:

‘We demonstrate a bandpass amplifier which can be constructed from common electronic components and has the surprising property that the group delay is negative in most spectral regions. A pulse propagating through a chain of such amplifiers is advanced by several milliseconds: the output waveform precedes the input waveform. Although striking, this behavior is not in conflict with causality, as demonstrated by experiments with pulses which start or end abruptly.’

(From https://www.physics.ohio-state.edu/~wilkins/writing/Handouts/abstracts_ajp.html)

I’m willing to bet that only those with a relatively advanced knowledge of physics will be able to derive much meaning from the above. A famous experiment along these lines is mentioned in Daniel Willingham’s Why Don’t Students Like School, in which a passage about baseball is given to pupils who have previously been tested on their knowledge of this specialised subject. The supposedly ‘poor readers’ with good knowledge of baseball understood the passage better, and did better on the test.

By far the most important component, therefore, of performance in reading tests is knowledge of the domain contained in the text to be read. This presents teachers with a problem, because they do not know which domain of knowledge will be important. The text could refer to anything, although test designers do try to avoid topics which are particularly obscure. In the absence of specific knowledge to be mastered, how can schools prepare pupils for reading tests most effectively?

The answer must be to teach as much knowledge and vocabulary as possible, and this is the responsibility of every teacher in the school. Every single lesson is preparing pupils for doing well in reading tests, if that lesson is building their knowledge and vocabulary. English teachers cannot take either the blame or the credit for the grades of pupils in their English Language GCSEs; most of the preparation has been done by others.

Given that building knowledge and vocabulary must be the priority, English departments must be realistic about what their role can be. The key knowledge which they can provide is knowledge of literature, but this is only one piece of the puzzle. Unless other departments (and primary schools) are providing a curriculum rich in knowledge, through proper teaching of a broad range of subject disciplines, many pupils will never perform very well in reading tests. Those who do particularly well without a knowledge rich curriculum succeed because of the knowledge and vocabulary they have acquired outside school.

Of course, outside the teaching of literature, there is something which English departments typically spend a large amount of time doing, especially as GCSEs approach: test drill. Many lessons are spent doing exercises that are more or less identical to the reading sections of the English Language examination: pupils are given passages they have never seen before, and required to answer a series of questions modelled on those posed by examination boards. Putting to one side the tedious and dull nature of this sort of exercise, which is likely to convince pupils that English is a very boring subject indeed, let’s ask whether these sorts of lessons are even likely to achieve their intended effect of increased GCSE grades.

I would argue that grade increases will be marginal, because the only likely benefit of this exercise is to build familiarity with the question types which pupils can expect to encounter. Pupils are very unlikely to acquire much new knowledge or vocabulary, because their working memory will be fully occupied by the simultaneous challenge of coping with an unseen text and answering questions on it. When the working memory is fully occupied with complex tasks like this, pupils can be working very hard for long periods of time and retain no new knowledge after the lesson, because their attention is completely taken up with the complexity of the task, and so there is no capacity remaining for transferring knowledge into long term memory.

The main result of spending large amounts of lesson time doing test drill is that pupils are more likely to perform at the level they were already at. In other words, they are more likely to be able to apply effectively the knowledge and vocabulary they already possessed. The lessons themselves are not giving them any new knowledge and vocabulary, and so are not actually increasing their capacity to read with understanding.

English teachers need to ask themselves what they hope to gain from lessons that mimic examinations. Examinations measure performance, but there is no expectation that they will increase performance, because there is no expectation that those taking the examination will acquire any new knowledge. One would hope, on the other hand, that our lessons actually taught our pupils something which they remembered.

Advertisements

English Language: The Vampire Subject

2000px-Little-vampire.svgMichael Fordham has already written eloquently about how English, as a whole, is a Frankenstein subject, a monster composed of a myriad of vastly different parts artificially stitched together. Within that jumble of odd parts, there are real academic disciplines that can be separated out. The key academic discipline, for the purposes of general education, is the study of literature.

By studying literature, our pupils have their minds opened upon other worlds; they encounter the great narratives that have shaped human thought and culture down the centuries; they gain cultural capital that will enable them to take part in educated reading and conversation for the rest of their lives. Studying literature is a hugely enriching part of every child’s education, and no one should be denied this opportunity.

But there is another GCSE which everyone takes: English language. What knowledge, precisely, does this test? In fact, it does not test anything in particular. It involves being given a more less random selection of extracts and writing tasks, that are supposed to be a test of the pupils’ general ability to read and write.

This might seem like a reasonable idea. Surely we want to know something about whether pupils are able to read with understanding and write accurately? Well, yes, but here’s the problem: once you get beyond the basics of decoding (in reading), and spelling and grammar (in writing), there’s no such thing as general reading and writing ability. Your ability to read and write well about any topic depends upon specific knowledge and vocabulary about that topic. Whatever topic happens to come up in the English language exam will be more accessible to those who happen to have more knowledge about that topic. General tests of reading and writing are always, inherently, unfair, because they can never take this into account.

So English language is a non-subject. If we want to find out whether pupils can read with understanding and write accurately, there is no reason why this should not be done through the vehicle of all the actual subjects that have real, specific substance. If we care about accurate writing and intelligent reading, then we need to make sure the GCSEs in literature, history, geography and science test these things. They can test them much more fairly than the English language exam, because the test will be based upon a specific body of knowledge which pupils are supposed to have mastered. No one will randomly gain an advantage because the test is based on randomly selected knowledge.

The Key Stage 2 Reading SATs suffer from exactly the same problem. They test reading in a general way, instead of basing the test upon specific knowledge which pupils are supposed to have mastered. Therefore, they are inherently unjust.

As well as the inherent unfairness of general tests in reading and writing, their existence and importance creates serious curricular misunderstandings. When high stakes tests are based around general reading, schools tend to spend lots of time teaching general reading, with endless comprehension worksheets and test drill. This has been happening in America for many years, where a very high stakes annual test of reading has led to a hollowing out of the curriculum, with the abandonment of real academic subjects which contain real substantive knowledge in favour of tedious lessons in how to ‘find the main idea’ in random bits and bobs of text. E D Hirsch has pointed out that these reading tests, which he previously supported, have had such a deleterious effect on the curriculum, that they are what he calls ‘consequentially invalid’ (see Chapter One of Why Knowledge Matters).

So as well as being a non-subject, English language (or English language arts, as it is called in the USA) is a vampire subject: it sucks the life out of the curriculum, emptying it of specific content in favour of teaching generic comprehension skills that don’t even exist, because comprehension is always based upon specific knowledge, and generic writing skills that don’t exist, because good writing is also based upon specific knowledge of the topic about which you are writing.

It doesn’t seem likely that the curse of general reading and writing tests is likely to be lifted any time soon, so we have to make the best of this bad situation. For a start, we could scrap most of the curriculum time currently devoted to the non-subject of English language and spend that time teaching pupils a meaningful body of knowledge within proper academic disciplines. If we want them to read and write well, we need to stop doing generic lessons on reading and writing, and do more and more on the rich knowledge and vocabulary that will actually enable good reading and writing. We’ll need a little exam drill in the final run up to GCSEs, because of their fiddly nature, but really, there shouldn’t be much need for English language at all once the basics of decoding (in reading) and spelling and grammar (in writing) are in place. There are effective programmes already in existence to train pupils in those areas, such as Expressive Writing. Implementing those will mean that there should be very little need for any other specific English language lessons at all, so we will be free to teach pupils specific, interesting, coherent content instead.

(Image from Wikimedia).

The Problems with Grade Forecasting

Pig

You don’t fatten a pig by weighing it.

‘We know what we are, but not what we may be’- Shakespeare, Hamlet

It has become a truism in education that good professionals are able to predict the grades of their pupils accurately. Like most truisms, this needs further examination.

If a running coach spends their time making trainees run race after race, and times each one, he will be able to predict their performance in the final competition very accurately. But does that mean he is a good running coach?

No. Good running coaches do not spend their time making trainees run race after race. They break down the complex skill of successfully running races into its component parts, and assign their trainees exercises to build up those component parts. Instead of wasting time having them run races so that they can measure their performance, they have trainees build stamina and speed through a carefully designed set of exercises. If they want to know how their trainees are progressing, they look at how they are improving in those exercises, because they know that improvement in those exercises will ultimately lead to a better final performance.

Likewise, good teachers do not waste time making pupils do the final performance so that they can measure their attainment in it. Instead, they break down that final performance into its component parts, and focus on improving pupils’ abilities in those. If it’s an essay on Macbeth that pupils need to write, the component parts consist of knowledge of the play’s plot, characters and themes, knowledge of the literary and dramatic techniques used by Shakespeare to communicate these, and knowledge of the historical background which informs the play, such as James I’s Daemonologie and the struggles for power in medieval Scotland.

Instead of wasting time making pupils write essay after essay on Macbeth, the effective teacher will build gradually towards that final performance with lots of low stakes testing on core knowledge. These tests will look very little like the final performance. They could be multiple choice questions on Jacobean beliefs about witchcraft, or oral quick-fire questioning on the staging of the banquet scene in Act Three, or tightly focused analytical paragraphs on the imagery of damnation in one of Macbeth’s soliloquies, or a myriad of other methods of formative assessment that both measure and embed the knowledge and skills required for the final performance.

None of these formative assessment tasks will generate data that is useful for forecasting grades. But all of them will be integrated into a programme of teaching which leads to an excellent formal performance. Thanks to the benefits of retrieval practice, every little test will make the knowledge that much more secure.

The teacher who uses these methods will be focused on actually teaching, not generating data. They will be focused on pupils, not spreadsheets. And they may well forecast final grades less accurately than another teacher who puts their pupils through endless repetitions of the final performance, in the mistaken belief that this is the best way of preparing for it.

The obsession with accurate grade forecasts is anti-educational. If managers think it allows them to target their interventions more accurately, they are wrong. The kind of assessment that makes forecasts more accurate is not the kind of assessment that builds a precise picture of where knowledge gaps are, so that teaching can be more finely tuned. A whole essay is too complex to allow teachers to gauge exactly where they need to reteach key knowledge.

So let’s drop this obsession and focus instead on making the grades as good as they can possibly be. Otherwise we’re just generating self-fulfilling prophecies and preening ourselves on our professional skill, while in reality we are limiting the attainment of our pupils.

Understanding or Memorising?

Herbert_simon_red_complete

Herbert A Simon (1916-2001)

One of the most damaging myths in education is that there is a conflict between memorisation and understanding, when in fact, they go together, and are both essential. This is one of the ideas demolished by Daisy Christodoulou in Seven Myths, under the heading of ‘facts prevent understanding’. The classic piece of research by Nobel laureate Herbert Simon on chess players shows how memorising and understanding are not in conflict; rather, they are intertwined and interdependent. Chess grand masters can look briefly at a chess board and then remember the positions of the pieces far better than those not skilled in chess. But they could only do this when the pieces were arranged as they would be in a game scenario. When they were randomly arranged, the chess players performed no better than the average.

The chess grand master’s supreme skill is based largely on memorising tens of thousands of game scenarios. Most of the time, most of what a grand master is doing in a chess game is remembering. This is not in conflict with understanding. Obviously the grand master understands the rules of chess, but then so does someone who is only a beginner. What distinguishes the grand master from the beginner is the difference between their long term memories. One is well-stocked with chess moves and scenarios, while the other isn’t. If the beginner wants to advance, she needs to put in the effort and get memorising. She needs to be drilled, or drill herself. She needs to study hard.

This applies to every academic subject. Understanding and memorisation are both essential, but the majority of the effort must go into memorising, because a pupil can grasp a concept readily in a lesson, but quickly forget it, because she has not been drilled in class, nor has she been required to self-quiz for homework to consolidate the knowledge, nor has she been tested at intervals to make sure it does not fade. Drilling, self-quizzing, testing at intervals: these are the foundations of teaching which enable everyone to make progress, because memorisation is absolutely essential, and it strengthens and consolidates understanding. Without the knowledge in your mind, how can you think about it? It is the lack of these foundations which leads to the lament which I have heard so often in so many staff rooms: “I taught them that material. Why can’t they remember it? Why did so many fail the end of term exam?”

It is because of the false distinction between understanding and memorisation that teachers do not focus anywhere near enough time on making sure the foundations are in place. The bulk of class and homework time needs to be dedicated to making sure that core knowledge is thoroughly mastered. If this is not done, then only the most able and motivated will make much progress. The most able and motivated may well then go on to become education professors, and because they were not required to memorise, they’ll think that everyone else can just blithely sail along without the hard work of deliberately committing key facts and procedures to memory.

This is known as expert blindness, and the more gifted a person is, the more likely they are to suffer from it. But all teachers suffer from it to some extent, and to overcome it, we need good programmes of instruction which emphasise drill and repeated practice. We can’t depend upon our gut feeling, or even our ‘professional judgement’, to know when something has been practised enough.

Further reading:

Rote Learning is Ace

Memory and Liberal Education

(Image from Wikimedia).

 

General Reading Tests Are Always Unfair

roger-moore

Should pupils be taught who this man is? Maybe . . .

General reading tests are always unfair because there is no such thing as general reading skill. Every text will be much clearer to those who have relevant background knowledge. Whatever text you choose, you will favour those pupils who know something about its topic beforehand. The test will therefore not be fair, because it will privilege those who just happen to know something about that subject.

The idea that you can create a general reading test which will fairly assess reading ability across the school population is based upon a false notion of what reading entails. Once you have mastered decoding, the most important factor is knowledge. You could be brilliant at reading texts about football, but hopeless at reading texts about politics.

E D Hirsch describes how several recent studies have conclusively shown that background knowledge is the number one factor in reading ability, trumping academic ability and IQ in pupils, and complexity in texts. In 1998 the Recht-Leslie study showed that ‘poor’ readers could read well when they were familiar with the topic, which was baseball. They ‘illustrated the general principle: when a topic is familiar, “poor” readers become “good” readers; moreover, when a topic is unfamiliar, normally better readers lose their advantage’ (Why Knowledge Matters, p88). In 1999 Schneider et al proved the same point, but with IQ, while in 2011 Arya et al showed that text complexity was relatively unimportant compared to domain knowledge (see Hirsch, p88-89). These three studies show that if you know about something, you will be a good reader of texts concerning that topic.

Now consider what happens in the English language GCSE, as currently constituted. It is entirely ‘unseen’, so specific preparation of knowledge is impossible. The reading texts could be about anything. Let’s say the examination board selects a text about spies. Some pupils will know a lot about spies. Some may even be spy geeks. Others will have no interest in the topic, and will not have developed much domain knowledge. The English teacher may have spent one or two lessons doing some reading about espionage and discussing the topic, or they may not. We can hardly blame them if they haven’t, as they had no idea that this topic, among the thousands of possibilities, would come up in the examination. The spy geeks will ace that test, while others’ performance will be based mostly on the random criterion of how much they happen to know about spies.

The pupils who do well will be congratulated on their performance, but to a large extent, they just got lucky. The English teachers may even be congratulated if they happened to be teaching a large number of spy geeks. They may even get a pay rise or promotion based on this coincidence.

This is completely unjust, both to pupils and to teachers. General examinations given to the whole population should always be based on specified content. The more specific the content is, the fairer the examination will be, as it will enable all pupils to master that knowledge, and help all teachers to teach it effectively. As well as being fairer, only testing specific topics would promote education as the transmission of knowledge, rather than the development of mythical generic skills.

Because of the absence of specific content, preparation for general reading tests ends up being largely a process of tedious drills in how to serve up to the examiner what they are looking for. ‘Make sure you mention structure’, advise the English teachers, or ‘look out for similes and metaphors, and make sure you comment on them.’ How dull, compared to the richness and variety of content on which we could be focusing. But such a box-ticking approach seems inevitable when faced with general reading tests, because the knowledge required has not been specified in advance.

But we must not lose hope. We must make things better despite the unfair assessment system which currently exists, and we must work to persuade policy makers that fairness and the promotion of knowledge depend upon making tests as specific as possible.

How can schools best cope with the situation until such a time as assessment policy catches up with the findings of cognitive psychology?

The answer must lie in giving as much coherent knowledge as possible to pupils, especially in the earlier years, and in spending the minimum amount of time on tedious drills in examination skills. We can also make sure that any internal tests which we set are based on specific content which pupils are expected to master.

Instruction in how to give the examiner what they want should be left to the months immediately prior to the exam. The rest of schooling should be devoted to making pupils as knowledgeable as possible, because if we want our pupils to be creative and to think critically, if we want them to read well and write well, it is knowledge which counts.

Further reading:

Against Analysis, or Why William Doesn’t Engage the Reader

Comprehension Worksh**ts

Liberation from Levels

800px-charlesdickens

This man wrote ‘effectively’. But was his writing ‘sophisticated’?

Since national curriculum levels were abandoned, schools have had to build their own systems for describing and assessing progress. Unfortunately, as with many of their new found freedoms, few schools are making the most of this opportunity to transform curriculum and assessment, and make them more specific, reliable and meaningful. Instead, they are producing schemes of work and assessment filled with recycled versions of the old national curriculum level descriptors.

It’s understandable that schools do not want to leave level descriptors behind. They were statutory for many years, and they are still used by examination boards to describe grades. The only problem is, they are nonsense, and educationally harmful nonsense, too.

For writing, they typically involve statements like ‘wide vocabulary’ or writing ‘effectively’, or writing with ‘sophistication’. What do these words actually mean? How ‘wide’ is ‘wide’? What does ‘sophisticated’ look like? Such statements do not provide an objective standard of measurement, because they are open to multiple interpretations.

A far better method of assessing writing is by ranking it. Systematised comparative judgement looks like an excellent way of harnessing the reliability of ranking. Teachers cannot agree consistently on a mark for any given piece of writing, but they can agree very consistently, when shown two pieces of writing, on which one is better.

Even in the absence of the more systematised comparative judgement, teachers can mark by ranking. When they have a set of essays or stories to mark, they can put them in a rank order, and if they have to assign a grade, they can do so according to that rank order.

An experienced marker is doing some kind of comparative judgement in any case. He will say ‘I know what an A looks like’. What this means, is that he has marked hundreds or even thousands of pieces of work and compared them mentally, placing A grade work at the top of the rank order. An experienced marker will never be thinking to himself, as he gazes at a mark scheme, ‘Now is this sophisticated, or is this effective?’ Even if he uses those words because the mark scheme tells him to, what he will really be thinking is, ‘Is this top, middle or bottom? How does this compare to the best writing I’ve seen?’ and similar questions.

As well as being hopeless for accurate assessment, level descriptors undermine efforts to build a knowledge curriculum. They tend to be generic. In the study of literature, they will contain statements like ‘Can analyse the author’s use of language’. If we’re measuring progress against vague, general statements like this, we will not be thinking in terms of content that must be mastered, but in terms of general skills that can supposedly be demonstrated with any content. The content becomes indifferent.

Level descriptors take us away from specific content, because they never say clear, specific things like ‘Knows in what century Chaucer lived’, or ‘Knows Chaucer’s attitude towards late medieval society’. In fact, it’s rare to see the word ‘knows’ in a level descriptor. They tend to be about what you can do, not what you know, and thus they perpetuate the myth of general skills divorced from specific domains of knowledge.

If we want to assess attainment meaningfully, and if we want to build a knowledge-rich curriculum focused on specific content, we need to consign level descriptors to educational oblivion. Once we’ve left them behind, we will be able to focus our attention on specific content that should be mastered, and measure attainment against exemplar work.

Specific content and exemplar work give teachers and pupils something clear and definite to aim for. Pupils will be relieved and pleased to be shown a definite way forward, instead of wandering in the vague territory of the difference between ‘effective’ and ‘sophisticated’. And teachers will be able to say to pupils, ‘I want you to know this’ and ‘I want you to be able to write like this’, instead of muddling around with grade boundaries and tick boxes.

Educational Perjury

Rubber StampWhen one of our daughters was in preschool, we were presented with a lovely file containing lots of labelled photographs. The photos combined with the annotations were supposed to ‘prove’ all kinds of things about our daughter’s development. The time taken to compile all of this must have been significant. At the same time, our daughter was not, as far as we could tell, being taught any kind of objective knowledge in a coherent systematic way. For example, she was encouraged to discover her own way of writing the letters of the alphabet, rather than being required to begin a systematic course that would have given her the first steps towards attractive and legible handwriting.

It was clear that more time was being spent gathering ‘evidence’ than doing explicit teaching. In spending their time this way, staff were dutifully following the requirements of the early years curriculum, as the school interpreted them.

Like joyriding through a multistorey carpark, this obsession with so-called ‘evidence’ is wrong on so many levels.

Firstly, the evidence isn’t even evidence. It is the recording of one current performance, not proof of any kind of fluent mastery. If a pupil knows something today, and we believe that means he will know it forevermore, we are deluding ourselves. Thus, if we spend time fabricating a wonderful one off performance for the sake of producing such spurious evidence, we are involved in nothing less than a time-wasting sham of education.

If current performance cannot determine mastery, does this mean that we should not be testing our pupils? Of course not. We should be testing our pupils very frequently. But the tests should be genuine tests, not fabricated performances, and we should not be abusing the results of the tests by claiming that they are evidence of mastery, or a lack of it.

Tests are an excellent way of consolidating knowledge that has already been taught. That is their key function. As a measure of mastery, they must be viewed with extreme suspicion. Even if everyone in the class scores 100℅ today, what will they score tomorrow? Or next month? We must always, always be on our guard against the widespread and pernicious fallacy that current performance is a measure of long term learning.

So how do we know when to move on? How can we ever have any confidence about whether our pupils have mastered something? Instead of generating spurious ‘evidence’ to ‘prove’ this, or just ‘going with our gut’, we need to turn to programmes of instruction that have been tried and tested over many years and with many pupils, and look at the ways they build mastery using a carefully designed schedule of spaced and interleaved repetition. We need to look at how programmes designed by organisations such as the National Institute for Direct Instruction work, and either adopt those programmes, if they focus on the content and skills we want to teach, or do our best to apply their principles of instruction to what we are teaching.

If we’re serious about mastery, we’ve got to stop wasting time faking evidence. It’s educational perjury.