Occasionally when I try to convince someone I am talking to that there is a logical flaw in their argument, I get a response along the lines of “I was not trying to be scientific, it’s weekend!”
I was just reading a blog post (in Dutch) by Jaap Donkers in response to critique on scores he provided of the quality of schools in the Netherlands, where he used predicted values of an explanatory regression model to correct national test scores for the demographic background of the school. I have no issues with much of the blog post, but this remark triggered my response: “The task of science is not exclusively to write good scientific articles or books. The task is also to be a service to society (…) With such service by scientists other criteria should be applied than for an article in a peer-reviewed journal” (my translation).
Both statements appear to be about the same thing: that scientific logic applies to science, but not outside, and, more importantly, that even when scientist contribute to society as scientists, that criteria are still different. This is certainly true in terms of the intelligibility of statements: academic jargon should be avoided in public discourse. But how about other scientific criteria?
Scientists criticize each other’s work on a variety of grounds, whereby some typical examples include criticizing the generalizability of claims (e.g. “the sample you used has characteristics that are not representative of the population as a whole, about which you draw conclusions”); criticizing the reliability of measures (e.g. when based on highly subjective judgments); pointing out confounding factors of a causal claim (e.g. “eating ice cream really does not increase the risk of drowning – it’s the weather!”); criticizing the validity of a measure (“what you say you are measuring is not actually what you measure”); etc.
Given these types of critique, it is highly doubtful whether scientists should apply different criteria outside academia. How is a claim that generalizes from an unrepresentative subsample to a population valid in public discourse while invalid in an academic article? Or a causal claim where there really is an alternative explanation? Or policy advice on the basis of statistics on invalid measures?
The issue is of course exacerbated when scientists are called in as scientists. Here the public tends to listen because these are not just numbers, but scientific numbers. They are not just claims or arguments, but they are statements made by a scientist. The audience attaches extra value to these statements exactly because they, by virtue of being expressed in the role of scientist, implicitly claim to be scientific. Surely then, the criteria that are normally applied to the evaluation of scientific argument should be strictly applied?
Of course, scientific criteria do not apply in all contexts. Art, intuition, etc. are alternative methods of acquiring knowledge of the world that are outside the scientific domain. But in many contexts, scientific criteria do apply, even when one does not attempt to be scientific (first example); and when someone acts in public in the role of scientists, certainly all criteria apply in full (second example).