White lab coats, mixing fluids or measuring weight and force are probably some common memories from experiments in high school science classes. Experiments are intrinsically linked to our understanding what science is and what scientists do. But what can researchers do when they cannot conduct experiments?
One solution, I will describe in this blog post, has been used in a paper published in 2016 on the effect of education on dementia risk. They found that every year of schooling reduces dementia risk by 1.1%. This sounds like exciting news, especially for all of us doing a PhD and spending 15+ years in school. But hold your horses, certainty of research results and reality in general is always more complicated.
Studying the effect of education on dementia is not an easy task and scientists (like me) focusing on this topic face many difficulties. For example, we go to school at young age and get dementia in old age, but with the contemporary dementia burden, we cannot start an experiment now and wait for 80 years to see what happens. And even if we did have all the time in the world, or could rewind time like Hermione in Harry Potter, very few of us would like to be told what to study and for how long. In other words, we would not like to be randomized to education. And researchers want to randomize in order to exclude other factors that could mask (confound) the relationship between education and dementia.
To overcome these problems, Nguyen and his research colleagues used an instrumental variable approach. This method is nowadays commonly used in economics and also promoted in epidemiology, for example by the Medical Research Council.
But what is this method about? The idea is quite simple. Instead of conducting an experiment and randomizing, the researchers look for a change in environment, for example policy change, that (hopefully) did the randomization for them. So instead of doing an experiment, you look at one nature/history conducted for you. And then use it as an “instrument” in your statistical analysis.
In this specific study, the researchers used a US state school policy changes as an instrument in their analysis. In theory, children characteristics should be independent of how the state they live in changes policy about mandatory enrollment age or earliest dropout age, which will affect school length. So if you calculate length of schooling based on the policies and compare children from different states with different policies you (hopefully) bypass the problem with other children’s characteristics (confounders) that could mask the relationship you are looking for.
Maybe you noticed that I wrote hopefully a few times in the previous paragraphs. What does science have to do with hope? Aren’t scientists supposed to find the TRUTH? If only it was that simple. When we study the real world, like in epidemiology, economics and social sciences, we are less sure about the conclusions we can draw based on the evidence in our studies.
Evidence is a buzzword of our societies nowadays. We want evidence-based medical interventions and policies. Yet, evidence is not everything as a recently published book Ending Medical Reversal by Prasad and Cifu (which I have not read yet) shows. According to the authors, approximately forty percent of newly introduced treatment procedures are not better than the old ones, and sometimes they are even worse.
Yet, there was evidence for introducing the new procedures in the first place. So how is this possible? Because there is not one type of evidence. The book and study methods were recently covered on EconTalk podcast. So if you want to know more about methods and evidence or you just wonder, why you are told every other week that different type of diet is the best one, I do recommend listening to the talk.
When we study the real, complex world we never get black and white answers. Those those wanting to implement have to decide when the gray answers are sufficiently good to be acted upon. And I, as a researcher in this field, am always faced with uncertainty regarding what I do and what I can conclude. Discomforting as it may be, certainty is not a property of our everyday reality. Yet, trying to understand the reality is a worthy endeavor, in my opinion.
In a recent post on this blog, Yossa wrote about the different types of biology. I would like to add that there are many different types of evidence. And different types of science, including science without experiments.