Classroom Research Is Junk Data

Most of the educational research that has ever been done has been conducted in real live situations: in other words, in schools and classrooms, involving real teachers and pupils. Sounds like a good idea, right? Because we need to see how ideas play out in the wild, yes?

No. Classrooms are a hopelessly ineffective place in which to conduct educational research. It is impossible to fulfil any of the conditions needed to establish scientific results with any useful degree of reliability.

It is impossible to establish a proper control, because the control will have to be an entirely different class with a different teacher. In other words, they will be so different that a comparison will not be valid. It is impossible to isolate any effect of the method being used from the general effect of receiving greater attention that always comes with participation in a trial. It is impossible to account for the effect of the teachers who have volunteered to take part in the trial, and are therefore by definition enthusiastic about its potential.

Classrooms are busy places full of distractions, and the teachers and the pupils experience all kinds of motivations and pressures that can never be accounted for in the supposed results of any trial. The slippery nature of the classroom environment, combined with the fact that, as David Didau has pointed out, you can’t observe learning anyway, only current performance, make classroom trials next to useless.

Trying to come to general conclusions about education from classroom trials is like trying to come to general conclusions about jurisprudence from observing the prison population.

This applies in spades to John Hattie’s conglomeration of such trials. It doesn’t matter how much data you crunch: if it’s all junk, then it’s a case of rubbish in, rubbish out, as the software engineers say.

It’s not as if we cannot discover anything about how learning happens. As a matter of fact, great progress has been made in the last century in understanding how we learn and how we forget. But these discoveries have not been made in classrooms. They have been made in carefully controlled experiments run by cognitive psychologists. We can also discover much about human learning through the use of reason, and reading philosophy, which aims to explain the human condition in general terms.

Of course, if you don’t believe in the existence of humanity, then you’ll struggle to learn much from either cognitive science or philosophy, because you have concluded that there is, in fact, nothing to learn. In which case, go ahead and perform any experiments you want on your tender charges. Who cares? They’re just a bunch of rats in the laboratory maze.

But for those of us who believe that the human race exists, and that reason and truth exist, it’s time we drew a sharp distinction between classroom practice, which practitioners endeavour to perfect in their specific circumstances, and general statements about how learning happens. We can learn from science (or, as it used to be known, natural philosophy), properly conducted, and we can learn from philosophy, properly understood.

What we learn from these sources is not likely to surprise us, unless we have been thoroughly indoctrinated in trendy pseudoscience that has supposedly been ‘proven’ in classroom trials.

UPDATE: Based on some very thoughtful and detailed comments from readers (including a series of tweets from @cbokhove), I have rethought the central argument of this post. I’ll leave it as it stands, along with the comments, as a record of a stimulating discussion (this is the power of social media!). But severe scepticism, rather than outright dismissal, should really be our attitude towards classroom based research (or, for that matter, any education research). Also, it should go without saying that the burden of proof rests with the person who proposes any kind of mandatory and/or time-consuming intervention based on the supposed evidence.


14 thoughts on “Classroom Research Is Junk Data

  1. These are some pretty bold claims that should be supported with evidence. You say that differences between teachers invalidates study however, that would only be the case if the differences between teachers were a very large amount of the reason a study found the effects it did. It might be that differences between teachers make little difference. Its possible that the similar things different teachers do, in a specific situation creates an effect. Its important not to decide what happens in a classroom before testing those assumptions.

    Also you say all the little stresses and particular circumstances make comparison invalid. However, all classrooms experience stresses and particular circumstances, so there is some kind of average amount of stresses and circumstances (albeit of different kinds) in common across classrooms. So if you compare to a large enough number of other classrooms you are comparing to some kind of average amount of stress.
    This is how medical trials work, pharmaceutical research often has no idea of all the physiological differences between patients they test drugs on, so they test a lot of people to get some kind of average of how most people are likely to react. There’s no point trying the control out all the complexity because the drug has to work in the real world, likewise ed research needs to test in real classrooms, sometimes referred to as ecological validity.


      • Are you saying pupils might deliberately behave differently if they know something about the way they are being taught is in a study. If so this seem a bit unlikely as doubtful primary age children would understand and unlikely secondary would care.


      • Surely it seems unlikely pupils would know or care that a program was being monitored for impact. As for teachers behaving differently its important to control for enthusiasts but that easy done by studying a cross section of teachers.
        These small challenges are hardly grounds for concluding all education research is futile.


  2. Along with following the evidence based research rather than on student observations in the classroom, comes along the requirement for standardized exams. A teacher’s assessment is fine, however there needs to be an ongoing necessity for exams to determine where improvement is required. And I am not talking about the high stakes performance tests here. I am talking about the regular exams which have gone the way of the dinosaur. Bring them back. And insist on them. It’s the only way to determine where our students are struggling, and to hold our education establishment accountable.


  3. Hi,

    There seems to be a central claim here that the only research worth doing is that which is generalisable. This reflects a worrying trend in the current push for evidence based practice which is to assume a positivist comparison between education and medicine.

    What if classroom research doesn’t seek the kind of upscaling that, for example, RCTs offer? What if classroom research is about what works for that teacher or/and those kids in that school? What if classroom research is about a deliberate reflective practice? What if classroom research is about ongoing professional dialogue which goes beyond “this lesson wasn’t outstanding because …”?

    “The body reacts to drugs without the involvement of the will. This cannot be said of a pupil’s reaction to teaching.”

    This is of course true, and precisely why we shouldn’t try to force educational research to behave like medical research. This is social science. We must be careful that in our drive for evidence based practice we don’t neglect the social.


    • I agree. I am heartily in favour of teachers thinking about their local problems and seeking the best local solutions. But this should not be called ‘research’. Teachers need to recognise their own dignity. They have reason and can resist so-called evidence based practice when it defies reason.


  4. I think that, if only one variable is being tested and if the data set is large enough, you should be able to run some kind of regression analysis in order to find whether the variable’s influence is statistically significant. When dealing with those pesky humans, it is impossible to account for and regulate every possible variable, but the whole point of statistics is to assign a calculation that best fits the situation and you’d be surprised at how the proper use of statistical analyses techniques can actually account for all kinds of potentially influential factors.

    What you do find in education, is a less-than-thorough approach to analyses of the effects of any particular ideology, method or whatever is being tested. The maths just isn’t good enough or there might not be any maths involved at all (where the researchers have simply ‘asked’ children what their feelings are about something). For example, when testing the effect of one particular thing, for example whether ‘use of iPads = higher levels of achievement’, I would have tried to identify as many influential factors as possible, run the experiment over a good long period of time, and then do a shit load of maths.

    I’m going to go out on a limb here and say that if you got a proper statistician or scientist with no alterior motive (like having something to sell, or believing some silly ideology) to design and run some aspect of educational research, the resulting data, analyses and conclusions would be far more reliable.

    It’s all about the maths.

    Liked by 1 person

  5. Pingback: Laboratory Schools | The Traditional Teacher

Thoughtful and reasonable discussion is always welcome.

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s