How Data Enchants School Leaders

Turkish_DelıghtThe so-called ‘data’ gathered from schools is hopelessly unreliable, because teacher assessment is inevitably biased, and even external assessments can never reliably assess learning, only performance on a given day.

This hopelessly unreliable data is then subjected to invalid methods of statistical analysis. Despite the fact that schools are not representative samples of the population, they are treated as if they were, and their targets are set and results judged accordingly.

So we start with unreliable data, then we push it through invalid analytical procedures, and the end result is lots of spreadsheets coloured in beautifully, which can be used by bureaucrats and school leaders to make their decisions.

When they gaze at these beautiful spreadsheets, managers and inspectors are little better than King Saul or Macbeth, off to consult the witches to find out how things will go.

People have always wanted to know more than they possibly can know, especially about what the future holds. Because their desire to know is so strong, they are in great danger of deluding themselves into thinking that they really do know. Gamblers are a case in point. If the roulette wheel came up black three times in a row, then they decide it will certainly come up red next time. They may even apply the principles of probability to their decisions to place a bet, forgetting that probability can never predict the future. Each spin of the wheel is unique and not influenced by previous events.

We know how addictive gambling is, and the sensation of power and control conferred by the beautiful spreadsheets is equally so. It is a kind of forbidden fruit, or, better, it is like the Turkish Delight offered to Edmund by the White Witch. It is a delightful confection which, once tasted, will be almost impossible to resist.

Gamblers pay for their hubris with their own financial ruin. But when school leaders and inspectors allow themselves to be deluded in similar ways, it is the lives of pupils that are ruined, and the careers of classroom teachers.

We need a few more Davids to stand up to the deluded arrogance of the Sauls. The sharp pebble in your sling will be the truth.

Further reading: Jack Marwood’s eloquent and devastating appendix to David Didau’s Wrong Book is a good place to start.

Advertisements

8 thoughts on “How Data Enchants School Leaders

  1. “Gamblers are a case in point. If the roulette wheel came up black three times in a row, then they decide it will certainly come up red next time. They may even apply the principles of probability to their decisions to place a bet, forgetting that probability can never predict the future. Each spin of the wheel is unique and not influenced by previous events.”

    Could you argue:

    Someone walks up to a roulette table believing it to be fair.
    Black comes up 10 times in a row.
    They change their betting accordingly because the outcomes of the spins strongly suggests that the table isn’t fair but is actually biassed towards black.
    So whilst each spin is unique the observations of spin outcomes on the table do give information about the table.

    Translate that to schools:

    You walk up to a school believing it to provide a good education
    10 years in a row the results are much poorer than would be expected of other similar schools(*)
    They change their view of the school becuase the outcomes of the exams strongly suggest that the school isn’t providing a good education
    so whuilst each set of results is unique the observations have given information about the quality of the school.

    (* note – I do take your point about the problems of reliability of internal assessments and the limitations of external assessments)

    Like

  2. I am presently asking FFT about their calculations. I am also requesting that my managers explain how the targets are derived. If more of us questioned the methods … (But I do understand how difficult it is to challenge management) in many institutions. Most managers do not have enough confidence in maths to explain the calculations so give it a try even if your maths is shaky. I second the vote for Jack Marwood. His site is http://icingonthecakeblog.weebly.com
    Look up the stuff on fft

    Like

  3. Just for absolute clarity FFT do not set ‘targets’ – they provide information to schools about how students with the same ks2 results have done nationally in exams in previous years.

    Eg. Steve got a high level 4 in Maths and a low level 5 in English. He is studying GCSE Geography. FFT tell us the actual achieved GCSE Geography results, looking across the nation as a whole, of students who had previously achieved a high 4 in Maths and a low 5 in English.

    Now some schools mangle this information and misapply the information and call it a target. That is the fault of schools rather than FFT who are very clear about that being bad practice.

    Like

    • You are correct in that they do stress they are not providing targets HOWEVER they do have some responsibility. They seem quite happy to take money from schools who all seem to feel they have to use this organization. I would argue they are complicit (if that’s the right word) in the problems of over accountability. A pub landlord who sells alcohol to someone he knows or strongly suspects is about to drink and drive has to take some responsibility (I am not a legal expert). Obviously FFT is the landlord. Data is the alcohol. Managers are the drunk drivers. Teachers are the victims of the inevitable RTC

      Like

  4. Pingback: Embracing Risk in a Climate of Fear and Surveillance | The Traditional Teacher

Thoughtful and reasonable discussion is always welcome.

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s