Post

More on Statistical Methods: Selection Bias (Francisco Ramirez, USA, 08/15/15 4:59 am)

Imagine that you go to the doctor and she prescribes medication to lower your bad cholesterol. You have the audacity to ask how she knows this medicine causes the bad cholesterol to get lower. She responds that there are studies that show that people who take this medication show decreases in their bad cholesterol more so than those who do not. You, a budding scientist, then ask but how do we know that the medicine takers did not also change their diets and increase their exercise. Furthermore, how do we know that the medicine takers did not differ from the others in other important respects to start with, e.g. better educated. That is, how do we know that the medicine is really the cause of the reduction? The doctor smiling responds that subjects were randomly assigned to the experimental condition where they got the medication or the control condition where they did not (maybe they got an alternative treatment). And you can read all about this in the New England Journal of Medicine! Your confidence increases and with good reason.

The problem with many studies that simply compare homeschoolers with non-homeschoolers on a number of outcomes is that there is no way of knowing whether the difference is due to the homeschooling experience or to other factors that may have given homeschoolers a selection advantage. So, if in the studies the parents of the home schoolers were better educated and had access to more resources than the parents of the schooled, these differences may be what truly results in better outcomes. It is the diet and the exercise and not the medication that lowered the cholesterol.

In more personal terms, I have no way of seriously knowing whether it is my mentoring that leads my advisees to educational and occupational success. These advisees were not randomly assigned to me. They were selected with less than 7 percent of the applicants offered entry. And, they were given more generous fellowships than in most other universities. So, how can I assert that my mentoring causes the favorable outcomes. I can point you to testimonials and to my mentoring awards. I can believe in myself and my judgments. But I know the problems with assuming that my mentoring is like the medication in an experimental study. I understand selection bias.

Therein lies one of the problems with all the studies that have been cited in favor of the thesis that homeschooling is on average better than other schooling. It may indeed work for the daughters of Enrique Torner. But our skepticism regarding the claim that on average homeschooling works better is based on our understanding of the uses and misuses of statistical analysis in social science research. Selection bias is a problem in a lot of research. On the other hand, selection bias is my best friend at Stanford!

JE comments:  This is an excellent explanation of selection bias.  Since they can benefit from his wisdom, Francisco Ramírez's students are truly the Select!

Visits: 1