Friday, May 10, 2013
Is John Cook planning to use systematically biased “correct” survey answers to make unbiased skeptics look biased?
Crossposted at WUWT (with some improvements here to the intro and postscript)
After finalizing a long
post on John Cook’s crowd-sourced consensus-rating survey (to be titled “I take
Cook’s survey so you don’t have to”), I submitted my completed survey to Cook’s
website and received an automated response that included a key new bit of
information, suggesting what likely shenanigan Cook has planned.
I am not going to
rewrite the post because it describes why I gave the ratings I did to each
abstract in the random survey that Cook’s website compiled for me. The likely
shenanigan has to do with how the rating rules are applied so I want it to be
clear that what I wrote on that subject was innocent of any awareness of how
Cook might be biasing the survey. I am just adding this brief introduction.
The new information
(new to me) is that Cook seems to be claiming to have in his back pocket a correct answer to what each of the
ratings should be. From the automated response:
Of the 10 papers that you rated, your average rating was 3.1 (to put that number into context, 1 represents endorsement of AGW, 7 represents rejection of AGW and 4 represents no position). The average rating of the 10 papers by the authors of the papers was 2.6.
It seems impossible
that Cook could actually have gotten large numbers of authors to apply his
rating scale to their papers. Maybe this is why he drastically reduced the
number of included papers from the 12,000 mentioned in the survey to only those
with abstracts shorter than 1000 characters (as discovered by Lucia at The Blackboard).
Maybe the full reduction is to papers that not only have short abstracts but
were also self-rated by authors. If so there is a clear selection bias and the
abstracts in Cook's sample are not representative of the literature [commenters
say that this limitation to self-rated abstracts has also been verified at The Blackboard].
Supposing that Cook
really does have author-ratings for all the papers used in the survey, there
is a major slip between the cup and the lip. The authors are described as
rating the papers, while
surveyors are asked to rate only the
abstracts. This is critical because according to Cook’s rating
rules the ratings are supposed to be based on what is or is not specifically
mentioned. Obviously full papers discuss a lot more things than abstracts,
especially unusually short abstracts. Thus if everyone is applying the rules
correctly surveyors ratings should be systematically higher (assessing less
conformity with consensus assumptions) than authors’ ratings.
This stood out to me
because I had just spent several hours describing how I had to rate abstract
after abstract "neutral," even though it clearly proceeded from
"consensus" presumptions, just because its abstract had not directly
mentioned those assumptions. The full papers might well have, making the author
ratings and the surveyor ratings apples and oranges.
Suppose (as is likely)
that survey participants who are referred by skeptic websites rate the
abstracts accurately according to the instructions while those who are referred
by credulous websites misapply the instructions so as to exaggerate the degree
of consensus. This misapplication of the rules will bring the ratings of the
consensoids closer to the ratings of the authors than the accurate ratings from
the skeptics will be, making the consensoid surveyors look less biased than the
skeptic surveyors when they are in fact more biased. Mission
My original post is
after the jump. It's point is rather different: that the revelation from the
survey is how neatly the various papers fall into simple categories of wasted
effort on the "consensus" plantation. In contrast, Cook's attempt to
gauge the degree of consensus turns out to be not very effective, which is another
reason (besides its coming from John Cook) why people shouldn't feel much need
take it.
I take Cook’s survey so you don’t have to
John Cook, creator of
the pathologically credulous Skeptical Science website, is asking
interested parties of all persuasions to participate in a crowd-sourced
assessment of the degree of consensus in the peer-reviewed literature on
climate change. What percentage of published papers affirm, the “consensus”
view that global warming has mostly been caused by human activity? How many
deny it, and how many abstain?
This is a question that
skeptics and believers are both interested in but for different reasons.
Believers use claims of consensus to support their arguments from authority:
who are you going to believe, all the scientists, or a few doubters who don’t
have sufficient training to properly grasp the issues?
Skeptics see the
“consensus” as manufactured by 20-plus years of politically allocated funding
and ideological bullying. The science itself is extremely uncertain and rife
with contra-indications, turning any high degree of conformity in the
peer-reviewed literature into a measure of intellectual corruption. (WUWT ran a
post on this the other week.)
Given Cook’s history of
shenanigans
there is concern that some kind subterfuge is planned, especially since he
asked different bloggers to post survey links with different tracking tags without
mentioning this in his invitation letter. That seems to have put a “hold” on
skeptic participation (WUWT, for one, is not
posting its unique survey link unless Cook can give a satisfactory
explanation).
I suggest a simple
condition for participation: that Cook promise to publish the full data, broken
down by abstract and binned by the skeptic vs. believer classifications of the
referrers. Cook’s rating system is reasonably unambiguous so where the
credulous and the skeptic sides differ in rating a particular abstract it will
be easy to see who is being honest.
I don’t think that even
the pathologically dishonest “consensus” side will find much to be dishonest
about here but the comparison obviously holds no threat for skeptics. Also, the
number of credulous and skeptic ratings for each abstract will allow the
randomness of the selection of the abstracts to be checked. The data could be
further broken down by referring blog but Cook did not advise invited blog
authors that this would be done so maybe that might be inappropriate at this
point, and it is not necessary for checking randomness.
Hopefully Cook will
agree to these terms, as his rating system does have some merit.
Cook does not try to
inflate “consensus” numbers by conflating the credulous and the skeptic
positions
That was the problem
with the bogus Doran
and Zimmerman survey that asked respondents whether they think CO2
emissions have a “significant” effect on climate. “Significance” is a
bottom-threshold criterion of undefined strength. To the extent that it has any
specific meaning it only refers to whether an effect is statistically
discernable, at however small a level. Few skeptics (being naturally modest in
their claims) would say that CO2 effects cannot even be called “significant,”
making Doran’s finding of 82% consensus meaningless. It’s a consensus on a
non-issue that skeptics and believers don’t disagree about.
The question is whether
CO2 emissions are dangerous,
and here Cook’s rating criterion is sound. For danger to even be a possibility
CO2 effects have to be larger than natural effects (which are almost certainly
headed in the cooling direction now that the sun has dropped into a quiescent
phase), and this is what Cook’s rating scale focuses on: to what extent does a
given abstract conform to the view that “humans are causing more than half of
global warming.”
One can still quibble.
By using the present tense (“humans are causing”) Cook presumably means to
refer only to the most recent global warming, post-1975 say. The planet cooled
for the 30 years before that and pretty much no-one believes that human CO2 emissions
had much impact before WWII, but Cook doesn’t let on how short a bout of
warming is being used to proclaim a whole new climate regime.
Use of the present
tense also presents global warming as an ongoing phenomenon, even though after
15 years of no statistically significant warming no one can avoid the question
of warming actually IS ongoing or whether it has stopped. (From Reuters
three weeks ago: “Climate scientists struggle to explain warming slowdown.”)
But these biases are
actually appropriate. The purpose of the survey is to gauge conformity with the
“consensus” position and the consensus, as articulated by the IPCC, is that
dangerous global warming is not just an ongoing problem but a rapidly growing
problem. Here Cook’s rating criterion—is warming mostly caused by people?—is
commendably straightforward, getting at the idea that human effects have
outstripped natural effects and we are now on some uncharted run-away course.
The last 15 years of no
discernable warming pretty much proves that human effects are not stronger than natural effects, at
least not yet, so it has to be swept under the rug. That’s the “consensus” way.
Might as well go ahead and measure it.
My survey experience
I wanted to see if
Cook’s survey exercise was at all revealing so I gave it a whirl, following the
link that he asked me to post on my Error
Theory blog.
A copy of the survey I
took is posted here. I
disabled the “submit” button, just in case it might actually work and mess up
Cook’s randomization, and I added the ErrorTheory-tagged survey link to the
top of the page, in case anyone really wants to send Cook some skeptic-tagged
responses. I don’t see any harm in it. If John tries to misuse the results that
just creates pressure for him to release the data (broken down by abstract)
which could be amusing.
The rating options
(more fully explained on the survey page) are:
1. Explicitly endorses and quantifies AGW as 50+%
2. Explicitly endorses but does not quantify or minimise
3. Implicitly endorses AGW without minimising it
4. Neutral
5. Implicitly minimizes/rejects AGW
6. Explicitly minimizes/rejects AGW but does not quantify
7. Explicitly minimizes/rejects AGW as less than 50%
8. Don’t know
As I detail below, all
ten of the abstracts that were selected for me to rate fit clearly within the
IPCC paradigm that humans are creating dangerous global warming. [This is a
little higher than I would expect on average, but it could be because the
abstracts are from an author-rated subset of the literature, which would not be
representative and might well tilt to the consensus side. (Comment added after
original post.)]
All but one (maybe two)
of the papers proceed from the assumption that big warming is coming, and since
nobody on either side of the debate is predicting substantial natural warming, the authors must be
implicitly assuming that the predicted warming will be human caused, but this
conformism doesn’t show up very well in Cook’s highly specific rating scheme.
Five of the articles
take the projections of big human caused warming so much for granted that they
don’t bother to mention the causes of warming at all, yielding a Cook-rating of
“4″ for “neutral” (“doesn’t address or mention [the] issue of what’s causing
global warming”). Further, none
are explicit in suggesting that people are causing most warming, making for a
modest average Cook-consensus-rating of 3.1 (less than one point from
“neutral”), despite the extreme degree of conformity exhibited by the papers.
So Cook’s rating scheme
does not provide a very good measure of consensus conformity, but it is revealing in a different way. The
exercise of classifying the abstracts give a pretty good picture of the various
spheres of meaningless toil that $100 billion in climate-science funding has
created.
How will an
ever-increasing rate of global warming harm x,y and z?
We are all familiar
with this category of conformist toiling and it is well represented in my
sample survey. (The links that follow are to the abstracts of the named
papers).
Why would anyone write
a paper on the “Possible Impact Of Global Warming On Cabbage Root Fly
(delia-radicum) Activity In The UK,” assessing an obscure possible consequence
of several degrees of global warming, if they didn’t believe the IPCC’s
projections that human CO2 emissions will indeed cause such warming? Okay,
there is another likely
reason: because politically funded climate science won’t support anything to do
with the natural causes of climate change, or the very real damage that even
mild global cooling would inflict. But regardless of whether the authors of the
paper are coercees or coercers, they are still part of the “consensus.”
In the same category
are: “Predicting Extinctions As A Result Of Climate Change,”
(based, of course, on the “consensus” predictions of climate change); and “Effect Of The Solid Content On Biogas Production From
Jatropha Curcas Seed Cake.” (Who would focus on horrendously inefficient
biofuels if they were not convinced that fossil fuels were killing the planet?)
Another article, “Changes
In Tropical Cyclone Number, Duration, And Intensity In A Warming Environment,”
finds a lead lining in the decreasing levels of hurricane energy. The number of
category 4 and 5 storms has gone up, they say, so we should still be alarmed.
(Just don’t tell ‘em what’s happened to global major
hurricane frequency since their paper came out in 2005, or accumulated
cyclone energy.)
As gung-ho as these
four sets of authors seem to be for the alarmist side, they don’t say anything
about why temperatures went up over the period they analyze, so the abstracts
have to be rated “neutral.”
Five of the papers are
about computer modeling of climate
Four of these simulate
GHG driven warming and should probably be rated a 2 (“Explicit Endorsement
without Quantification: abstract explicitly states humans are causing global
warming or refers to anthropogenic global warming/climate change as a given
fact.”)
One, for instance,
predicts, as stated in its title: “Intensified Asian Summer Monsoon And Its Variability In A
Coupled Model Forced By Increasing Greenhouse Gas Concentrations.” The model
results actually would yield a quantification of the GHG effects on warming and
variability, and since these models don’t anticipate secular changes in any
forcings other than GHGs, they presumably end up attributing essentially all
predicted climate change to rising CO2. Still, the abstract doesn’t actually
say anything about current warming being caused more than 50% by CO2 so the
paper can’t be rated a 1. Ditto for “Projection of Future Sea Level,” “Dynamic And Thermodynamic Influences On Intensified Daily
Rainfall,” and “Changes
in Water Vapor Transport.”
None of these model
simulations provide any evidence
for CO2 driven warming. The models used are all calibrated using the assumption that no other important
forcings were at work over the 20th century. In particular, they all assume
there is no mechanism of solar forcing beyond the very slight variation in
total solar irradiance, which is parameterized in current IPCC models as having
only 1/40th the forcing effect of increasing CO2 (AR5 SOD p. 8-39 table 8.7).
Since these models
start with the assumption that warming is caused by CO2 they cannot in turn
provide evidence for that proposition. That would be a circular argument. The
one exception might be if the authors could make a solid case that their models
succeed in capturing complex emergent phenomena observed in nature, but this is
the opposite of what is actually happening. The models are completely missing
on their basic predicted “fingerprints,” like the predicted upper-tropospheric
“hotspot” that thermometers can’t find.
The fifth computer
model paper (“Scaling in the Atmosphere”) looks at how to achieve higher
resolution models. Since all the big taxpayer-funded computer models are driven
primarily by CO2 and completely omit any enhanced solar forcing, efforts to
refine the resolution of these models is another form of toiling in the
“consensus” quarry, yet this paper has to be Cook-rated “neutral” because it
doesn’t even mention any predicted warming, never mind its cause.
That’s five “neutral”
ratings and four “explicit without quantification” out of nine thoroughly
consensus-conforming papers.
The last abstract is
the only one that addresses the evidence for what might have caused the modicum
of late 20th century warming, and it is the most revealing
By coincidence, this
happens to be a paper I wrote about in 2011. It is Knud Lassen’s 1999 update to his seminal 1991
paper with Friis-Christensen. Their original research had found a strong
correlation between solar cycle length and global temperature change:
Evidence of a
substantial solar effect on climate would soon come to be regarded as
“contrarian” and Lassen at least seems to have been eager to get over to the
safe side and renounce his earlier work. When the 1997-98 El
Nino caused global temperatures to spike dramatically Lassen was
hair-trigger on the opportunity to issue a 1999 update, declaring that with the
rapid increase in recent temperatures the correlation between solar cycle
length and temperature had been broken.
This really was in
reaction just to the El Nino spike. There is no way that Lassen could have said
in 1996 that the planet had warmed too much to consist with the pattern he and
Friss-Christensen had discovered through 1991. Solar cycle 22, which ended in
1996, was the shortest
of the modern-maximum era so Lassen should at that point have been expecting
temperatures to rise, but take a look at the satellite temperature record
through 96. Unless you want to cherry pick end points there is no significant
trend going back from 96 to the beginning of the record:
Before the El Nino,
Lassen should have been seeing temperatures as below what his 1991 paper would predict. Only the 97-98 El
Nino offered Lassen a data-based opportunity to bail on his now-contrarian
earlier work and claim that temperatures had become too high, and he jumped
right on it, even though the whole world knew that the temperature spike was
due to an ocean oscillation and hence would be at least largely temporary.
This was a desperately
unscientific move. While Lassen was still writing his update in mid ’99
temperatures had already dropped back to 1990 levels. His eight year update was
outdated before it was published.
So how eager has Lassen
been to update again, now that the next dozen years of data have strongly
supported the solar warming theory? Crickets. The long solar cycle 23 has
coincided with at least an extended pause in global warming and Mr.
rush-to-update is nowhere to be found.
Bah, who needs him?
Others have gone on to discover that the stronger correlation is between solar
cycle length and global temperature change over the subsequent
solar cycle, suggesting that the planet may now be heading into a cooling
phase. That’s actually worth worrying about, but some people can only worry
about fitting in.
The rating for Lassen,
even though he is the most obsequious out of ten obsequious sets of
consensus-conforming authors, doesn’t rate particularly high on Cook’s
consensus scale. That’s because he doesn’t actually mention GHGs at all, but
just claims that solar cycle length seems not to have been the cause of the
1990s warming, give him a 3:
Implicit Endorsement: abstract implies humans are causing global warming. E.g., research assumes greenhouse gases cause warming without explicitly stating humans are the cause.
Summary
Using John Cook’s
highly specific criteria for identifying supporters of the “consensus” that
people are causing dangerous global warming, my final tally is five abstracts
with a rating of 4 (“neutral”), four with a rating of 2 (“explicit endorsement
without quantification”), and one with a rating of 3 (“implicit endorsement”),
yielding an average rating of 3.1 out of 7.
My broader assessment
is that all the authors are flaming full-on consensoids. Mr. Cook is welcome to
avail himself of this assessment if he thinks it strengthens the case for a
genuine scientific consensus. I have a different interpretation. My anecdotal
sampling, if it turns out to be representative, strongly supports the skeptical
charge that climate science is thoroughly dominated by a tyrannical
politically-fabricated and monetarily-enforced “consensus.”
Almost all the papers
in my sample simply assume that big fat warming, implicitly anthropogenic is on
the way, either building computer models based on this assumption, or
projecting its various possible harmful consequences. Not one of the ten
contains even a shred of evidence
for the CO2-warming theory. The one paper that pretends to provide evidence
against the competing solar-warming theory was a mad rush to misrepresent a
temperature spike that the authors had every reason to believe was the product
of ocean oscillations and they have since proven their bias by refusing to
acknowledge that the much longer subsequent record of no warming strongly
supports their original hypothesis of solar-driven climate change.
In all, a clear picture
of ideological bullying, self-censorship and rent-seeking, exactly what we
should expect from a politically created and controlled branch of science.
Postscript
After being frustrated
by how often the abstracts did not contain the statements necessary to rate
them as consensus-conforming, even though they obviously were, it was obvious
that the results might have been different if I was rating full the papers.
Thus I was startled to see Cook not just let on that he had author ratings for
the papers, but to see him comparing my rating of the abstracts directly to
these ratings of the papers, as if the two should be comparable.
Obviously the author
ratings should be lower, making mine look "wrong" and perhaps biased,
as if I was reluctant to concede how consensus-conforming the papers were. No
way. I was frustrated at not being able to acknowledge how consensus-conforming
the papers were, making Cook's apples and oranges comparison look like obvious
trickery. Is it Cook’s plan to invoke the authors’ self-ratings as an accurate
standard of what the ratings should be? He seems to already be doing this in
the automated feedback that he returns to respondents when surveys are
submitted.
It is possible that he plans to note in his
write-up how the author ratings, being ratings for papers not abstracts, should
show a systematically higher level of consensus conformity (a lower numerical
rating) than an honest and careful surveyor’s ratings of the abstracts for the
same papers. We’ll see, but I suspect that Cook is instead planning on using
the systematically high author ratings to accuse participants from skeptic
blogs of bias for estimating less consensus conformity than the authors
themselves.
At the same time he can
count on participants from credulous blogs to overrate the degree of consensus
conformity, making them look less biased and more honest than skeptical
participants when they are actually more biased and less honest. As we all
know, and as Cook would know, there is widespread belief amongst climate
alarmists that it is okay to misrepresent specifics in support of the “larger
truth” that human impacts on the planet need to be dramatically curtailed.
According to the late Stephen
Schneider, one of the founding fathers of climate alarmism:
Each of us has to decide the right balance between being effective and being honest.
Participants from both
sides will feel an urge to rate the abstracts as more consensus-conforming than
the rating instructions say, just because the instructions are so conservative
in the false-positive-avoiding sense. Paper after paper that has obviously
sprung from the “consensus” womb is to have its abstract rated “neutral.” Both
sides will also feel a partisan
interest in registering this actual degree of consensus conformity, with
believers wanting to bolster their arguments from authority and skeptics being
glad to document the blatantly unscientific groupthink in the peer-reviewed
literature. But unlike skeptics, the consensus side believes that dishonesty in
public debate can be both moral and efficacious, and Cook can use that.
One indication that he
is thinking in this direction is the failure to mention in his invitation
letter that different survey links were provided to each invitee. This omission
suggests that he was trying to get away with something. In a court of law such
behavior is taken as an indicator of “guilty conscience.” So what was his
sneaky purpose?
This is indicated by
the second thing he is not upfront about. He cites what would seem to be a
“correct” estimate of what the ratings should be without noting that the
self-estimates from the scientific authors are systematically biased vis a vis the survey questions. That
extends the pattern of surreptitiousness and it suggests the purpose. Yeah, he
was going to use this known-to-be-wrong “correct” rating to make the dishonest
ratings from his consensoid compatriots look honest and skeptic ratings look
biased.
He might not be able to
get away with it now, but it
is reasonable to suspect that this was
the plan. Still, Cook’s effort need not be for nothing. The fact that the
author ratings of the papers
should be numerically lower than the ratings for the abstracts provides a simple test for
which group of survey participants is less biased. The group that is more
honest and self-controlled, giving in less to confirmation bias, will submit
lower assessments of the degree of consensus conformity (numerically higher
ratings).
In particular, these
rating should be numerically high compared to the author’s self-ratings, so
whichever group comes out closest to the author ratings should be more biased
and less honest. I’d like to see that experiment go through, though it might be
tainted now that people know Watts
Up.
Comments:
<< Home
I really enjoyed reading your articles. This is truly a great read for me. I have bookmarked it and I am looking forward to reading new articles. Keep up the good work.
Post a Comment
<< Home
