Most of the educational research that has ever been done has been conducted in real live situations: in other words, in schools and classrooms, involving real teachers and pupils. Sounds like a good idea, right? Because we need to see how ideas play out in the wild, yes?
No. Classrooms are a hopelessly ineffective place in which to conduct educational research. It is impossible to fulfil any of the conditions needed to establish scientific results with any useful degree of reliability.
It is impossible to establish a proper control, because the control will have to be an entirely different class with a different teacher. In other words, they will be so different that a comparison will not be valid. It is impossible to isolate any effect of the method being used from the general effect of receiving greater attention that always comes with participation in a trial. It is impossible to account for the effect of the teachers who have volunteered to take part in the trial, and are therefore by definition enthusiastic about its potential.
Classrooms are busy places full of distractions, and the teachers and the pupils experience all kinds of motivations and pressures that can never be accounted for in the supposed results of any trial. The slippery nature of the classroom environment, combined with the fact that, as David Didau has pointed out, you can’t observe learning anyway, only current performance, make classroom trials next to useless.
Trying to come to general conclusions about education from classroom trials is like trying to come to general conclusions about jurisprudence from observing the prison population.
This applies in spades to John Hattie’s conglomeration of such trials. It doesn’t matter how much data you crunch: if it’s all junk, then it’s a case of rubbish in, rubbish out, as the software engineers say.
It’s not as if we cannot discover anything about how learning happens. As a matter of fact, great progress has been made in the last century in understanding how we learn and how we forget. But these discoveries have not been made in classrooms. They have been made in carefully controlled experiments run by cognitive psychologists. We can also discover much about human learning through the use of reason, and reading philosophy, which aims to explain the human condition in general terms.
Of course, if you don’t believe in the existence of humanity, then you’ll struggle to learn much from either cognitive science or philosophy, because you have concluded that there is, in fact, nothing to learn. In which case, go ahead and perform any experiments you want on your tender charges. Who cares? They’re just a bunch of rats in the laboratory maze.
But for those of us who believe that the human race exists, and that reason and truth exist, it’s time we drew a sharp distinction between classroom practice, which practitioners endeavour to perfect in their specific circumstances, and general statements about how learning happens. We can learn from science (or, as it used to be known, natural philosophy), properly conducted, and we can learn from philosophy, properly understood.
What we learn from these sources is not likely to surprise us, unless we have been thoroughly indoctrinated in trendy pseudoscience that has supposedly been ‘proven’ in classroom trials.
UPDATE: Based on some very thoughtful and detailed comments from readers (including a series of tweets from @cbokhove), I have rethought the central argument of this post. I’ll leave it as it stands, along with the comments, as a record of a stimulating discussion (this is the power of social media!). But severe scepticism, rather than outright dismissal, should really be our attitude towards classroom based research (or, for that matter, any education research). Also, it should go without saying that the burden of proof rests with the person who proposes any kind of mandatory and/or time-consuming intervention based on the supposed evidence.