29. May 2012 · 5 comments · Categories: education

A few people today have pointed me at the new paper by Dan Kahan & colleagues (1), which explores competing explanations for why lots of people don’t regard climate change as a serious problem. I’ve blogged about Dan’s work before – their earlier studies are well designed, and address important questions. If you’re familiar with their work, the new study isn’t surprising. They find that people’s level of concern over climate change doesn’t correlate with their level of science literacy, but does correlate with their cultural philosophies. In this experiment, there was no difference in science literacy between people who are concerned about climate change and those who are not. They use this to build a conclusion that giving people more facts is not likely to change their minds:

A communication strategy that focuses only on transmission of sound scientific information, our results suggest, is unlikely to do that.

Which is reasonable advice, because science communication must address how different people see the world, and how people filter information based on their existing worldview:

Effective strategies include use of culturally diverse communicators, whose affinity with different communities enhances their credibility, and information-framing techniques that invest policy solutions with resonances congenial to diverse groups

Naturally, some disreputable newsites spun the basic finding as a “Global warming skeptics as knowledgeable about science as climate change believers, study says“. Which is not what the study says at all, because it didn’t measure what people know about the science.

The problem is that there’s an unacknowledged construct validity problem in the study. At the beginning of the paper, the authors talk about “the science comprehension thesis (SCT)”:

As members of the public do not know what scientists know, or think the way scientists think, they predictably fail to take climate change as seriously as scientists believe they should.

…which they then claim their study disproves. But when you get into the actual study design, they quickly switch to talking about science literacy:

We measured respondents’ science literacy with National Science Foundation’s (NSF) Science and Engineering Indicators. Focused on physics and biology (for example, ‘Electrons are smaller than atoms [true/false]’; ‘Antibiotics kill viruses as well as bacteria [true/false]’), the NSF Indicators are widely used as an index of public comprehension of basic science.

But the problem is, science comprehension cannot be measured by asking a whole bunch of true/false questions about scientific “facts”! All that measures is the ability to do well in a pub trivia quizzes.

Unfortunately, this mistake is widespread, and leads to an education strategy that fills students’ heads with a whole bunch of disconnected science trivia, and no appreciation for what science is really all about. When high school students learn chemistry, for example, they have to follow recipes from a textbook, and get the “right” results. If their results don’t match the textbooks, they get poor marks. When they’re given science tests (like the NSF one used in this study), they’re given the message that there’s a right and wrong answer to each question, and you just gotta know it. But that’s about as far from real science as you can get! It’s when the experiment gives surprising results that the real scientific process kicks in. Science isn’t about getting the textbook answer, it’s about asking interesting questions, and finding sound methods for answering them. The myths about science are ground into kids from an early age by people who teach science as a bunch of facts (3).

At the core of the problem is a failure to make the distinction that Maienschein points out between science literacy and scientific literacy (2). The NSF instrument measures the former. But science comprehension is about the latter – it…

…emphasizes scientific ways of knowing and the process of thinking critically and creatively about the natural world.

So, to Kahan’s interpretation of the results, I would add another hypothesis: we should actually start teaching people about what scientists do, how they work, and what it means to collect and analyze large bodies of evidence. How results (yes, even published ones) often turn out to be wrong, and what matters is the accumulation of evidence over time, rather than any individual result. After all, with Google, you can now quickly find a published result to support just about any crazy claim. We need to teach people why that’s not science.

Update (May 30, 2012): Several people suggested I should also point out that the science literacy test they used is for basic science questions across physics and biology; they did not attempt to test in any way people’s knowledge of climate science. So that seriously dents their conclusion: The study says nothing about whether giving people more facts about climate science is likely to make a difference.

Update2 (May 31, 2012): Some folks on Twitter argued with my statement “concern over climate change doesn’t correlate with … level of science literacy”. Apparently none of them have a clue how to interpret statistical analysis in the behavioural sciences. Here’s Dan Kahan on the topic. (h/t to Tamsin).


(1) Kahan, D., Peters, E., Wittlin, M., Slovic, P., Ouellette, L., Braman, D., & Mandel, G. (2012). The polarizing impact of science literacy and numeracy on perceived climate change risks Nature Climate Change DOI: 10.1038/nclimate1547
(2) Maienschein, J. (1998). Scientific Literacy Science, 281 (5379), 917-917 DOI: 10.1126/science.281.5379.917
(3) William F. McComas (1998). The Principle Elements of the Nature of Science: Dispelling the Myths The Nature of Science in Science Education

I’ve been following a heated discussion on twitter this past week about a planned protest on Sunday in the UK, in which environmentalists plan to destroy a crop of genetically modified wheat being grown as part of a scientific experiment at Rothamsted, in Hertfordshire (which is, incidentally, close to where I grew up). Many scientists I follow on twitter are incensed, calling the protest anti-science. And some worry that it’s part of a larger anti-science trend in which the science on issues such as climate change gets ignored too. In return, the protesters are adamant that the experiment should not be happening, no matter what potential benefits the research might bring.

I’m fascinated by the debate, because it seems to be a classic example of the principle of complementarity in action, with each group describing things in terms of different systems, and rejecting the others’ position because it makes no sense within their own worldview. So, it should make a great case study for applying boundary critique, in which we identify the system that each group is seeing, and then explore where they’ve chosen to draw the boundaries of that system, and why. I think this will make a great case study for my course next month.

I’ve identified eight different systems that people have talked about in the debate. This is still something of a work in progress (and I hope my students can extend the analysis). So here they are, and for each some initial comments towards a boundary critique:

  1. A system of scientists doing research. Many scientists see the protests as nothing more than irrational destruction of research. The system that motivates this view is a system of scientific experimentation, in which expert researchers choose problems to work on, based on their expectation that the results will be interesting and useful in some way. In this case, the GM trials are applied research – there is an expectation that the modified wheat might lead to agricultural improvements (e.g. through improved yield, or reduced need for fertilizer or pesticide). Within this system, science is seen as a neutral pursuit of knowledge, and therefore, attempts to disrupt experiments must be “anti-knowledge”, or “anti-science”. People who operate within this system tend to frame the discussion in terms of an attack on a particular group of researchers (on twitter, they’ve been using the hashtag #dontdestroyresearch), and they ask, pointedly, whether green politicians and groups condone or condemn the destruction. (The irony here is that the latter question itself is, itself, unscientific – it’s a rhetorical device used in wedge politics – but few of the people using it acknowledge this). Questions about whether certain kinds of research are ethical, or who might yield the benefits from this research lie outside the boundary of this system, and so are not considered. It is assumed that the researchers themselves, as experts, have made those judgments well, and that the research itself is not, and cannot be, a political act.
  2. A system of research ethics and risk management. If we expand the boundaries of system 1 a little, we see a system of processes by which scientific experiments are assessed for how they manage the risks they pose to be public. Scientific fields differ in their sophistication for how they arrange this system. In the physical sciences, the question often doesn’t arise, because the the research itself carries no risk. But in medical and social sciences, processes have arisen for making this judgement, sometimes in response to a disaster or a scandal. Most research institutions have set up Internal Review Boards (IRBs) who must approve (or prevent) research studies that poses a risk to people or ecosystems. My own research often strays into behavioural science, so I frequently have to go though our ethics approval process. The approvals process is usually frustrating, and I’m often surprised at some of the modifications the ethics board asks me to make, because my assessment of the risk is different to theirs. However, if I take a step back, I can see that both the process and the restrictions it places on me are necessary, and that I’m definitely not the right person to make judgements about the risks I might impose on others in my research. The central question is usually one of beneficence: does the value of the knowledge gained outweigh any potential risk to participants or others affected by the study? Some research clearly should not happen, because the argument for beneficence is too weak. In this view, the Rothamsted protest is really about democratic control of the risk assessment process. If all stakeholders aren’t included, and the potential impact on them is not taken seriously, they lose faith in the scientific enterprise itself. In the case of GMOs, there’s a widespread public perception (in the UK) that the interests of large corporations who stand to profit from this research are being allowed to drive the approvals process, and that the researchers themselves are unable to see this because they’re stuck in system 1. I’ve no idea how true this is for GMO research, but there’s plenty of evidence that’s it’s become a huge problem in pharmaceutical research. Medical research organizations have, in the last few years, taken significant steps to reduce the problem, e.g by creating registers of trials to ensure negative results don’t get hidden. The biotech research community appear to be way behind on this, and much research still gets done behind the veil of corporate secrecy. (The irony here is that the Rothamsted trials are publicly funded, and results will be publicly available, making it perhaps the least troublesome biotech research with respect to corporate control. However, that visibility makes it an easy target, and hence, within this system, the protest is really an objection to how the government ran the risk assessment and approval process for this experiment).
  3. A system of ecosystems and contaminants that weaken them. Some of the protesters are focused more specifically on the threat that this and similar experiments might pose on neighbouring ecosystems. In this view, GMOs are a potential contaminant, which, once released into the wild cannot ever be recalled. Ecosystems are complex systems, and we still don’t understand all the interactions that take place within them, and how changing conditions can damage them. Previous experimentation (e.g. the introduction of non-native species, culls of species regarded as pests, etc), have often been disastrous, because of unanticipated system interactions. Within this system, scientists releasing GMOs into the wild are potentially repeating these mistakes of the past, but on a grander scale, because a GMO represents a bigger step change within the system than, say, selective breeding. Because these ecosystems have non-linear dynamics, bigger step changes aren’t just a little more risky than small step changes; they risk hitting a threshold and causing ecosystem collapse. People who see this system tend to frame the discussion in terms of the likelihood of cross-contamination by the GMO, and hence worry that no set of safeguards by the researchers is sufficient to guarantee the GMO won’t escape. Hence, they object to the field trials on principle. This trial is therefore, potentially, the thin end of the wedge, a step towards lifting the wider ban on such trials. If this trial is allowed to go ahead, then others will surely follow, and sooner or later, various GMOs will escape with largely unpredictable consequences for ecosystems. As the GMOs are supposed to have a competitive advantage of other related species, once they’ve escaped, they’re likely to spread, in the same way that invasive species did. So, although the researchers in this experiment may have taken extensive precautions to prevent cross-contamination, such measures will never be sufficient to guarantee protection, and indeed, there’s already a systematic pattern of researchers underestimating the potential spread of GMO seeds (e.g. through birds and insects), and of course, they routinely underestimate the likelihood of human error. Part of the problem here is that the researchers themselves are biased in at least two ways: they designed the protection measures themselves, so they tend to overestimate their effectiveness, and they believe their GMOs are likely to be beneficial (otherwise they wouldn’t be working on them), so they downplay the risk to ecosystems if they do escape. Within this system, halting this trial is equivalent to protecting the ecosystems from risky contamination. (The irony here is that a bunch of protesters marching into the field to destroy the crop is likely to spread the contamination anyway. The protesters might rationalize it by saying this particular trial is more symbolic, because the risk from any one trial is rather low; instead the aim is to make it impossible for future trials to go ahead)
  4. A system of intellectual property rights and the corresponding privatization of public goods. Some see GMO research as part of a growing system of intellectual property rights, in which large corporations gain control of who can grow which seeds and when. In Canada, this issue became salient when Monsanto tried suing farmers who were found to have their genetically modified corn planted in their fields, despite the fact that those farmers had never planted them (it turned out the seeds were the result of cross-contamination from other fields, something that Monsanto officially denies is possible). By requiring farmers to pay a licence fee each year to re-plant their proprietary seeds, these companies create a financial dependency that didn’t exist when farmers were able to save seeds to be replanted. Across developing countries, there is growing concern that agribusiness is gaining too much control of local agriculture, creating a market in which only their proprietary seeds can be planted, and hence causing a net outflow of wealth from countries that can least afford it to large multi-national corporations. I don’t see this view playing a major role in the UK protests this week, although it does come up in the literature from the protest groups, and is implicit in the name of the protest group: Take The Flour Back.
  5. An economic system in which investment in R&D is expected to boost the economy. This is the basic capitalist system. Companies that have the capital invest in research into new technologies (GMOs) that can potentially bring big returns on investment for biotech corporations. This is almost certainly the UK government’s perspective on the trials at Rothamsted – the research should be good for the economy. It’s also perhaps the system that motivates some of the protesters, especially where they see this system exacerbating current inequalities (big corporations get richer, everyone else pays more for their food). Certainly, economic analysis of the winners and losers from GM technology demonstrate that large corporations gain, and small-scale farmers lose out.
  6. A system of global food supply and demand, in which a growing global population, and a fundamental limit on the land available for agriculture, place serious challenges on how to achieve a better match of food consumption to food production. In the past, we solved this problem through two means: expanding the amount of land under cultivation, and through the green revolution, in which agricultural yields were increased by industrialization of the agricultural system and the wide-scale use of artificial fertilizers. GMOs are (depending on who you ask) either the magic bullet that will allow us to feed 9 billion people by mid-century, or, more modestly, one of many possible solutions that we should investigate. In this system, the research at Rothamsted is seen as a valuable step towards solving world hunger, and so protesting against it is irrational. The irony here is that improving agricultural yields is probably the least important part of the challenge of feeding 9 billion people: there is much more leverage to be had in solving problems of food distribution, reducing wastage, and reducing the amount of agricultural land devoted to non-foods.
  7. A system of potential threats to human health and well-being. Some see GMOs as a health issue. Potential human health effects include allergies, and cross-species genetic transfer, although scientists dismiss both, citing a lack of evidence. While there is some (disputed) evidence of such health risks already occurring, on balance this is more a concern about unpredictable future impacts, rather than what has already happened, which means an insistence on providing evidence is irrelevant: a bad outcome doesn’t have to have already occurred for us to take the risk seriously. If we rely on ever more GMOs to drive the global agricultural system, sooner or later we will encounter such health problems, most likely through increased allergic reaction. Allergies themselves have interesting systemic properties – they arise when the body’s normal immune system, doing it’s normal thing, ends up over-reacting to a stimulus (e.g. new proteins) that is otherwise harmless. The concern here, then, is that the reinforcing feedback loop of ever more GM plant variants means that, sooner or later, we will cross a threshold where there is an impact on human health. People who worry about this system tend to frame the discussion using terms such as “Frankenfoods“, a term that is widely derided by biotech scientists. The irony here is that by dismissing such risks entirely, the scientists reduce their credibility in the eyes of the general public, and end up seeming even more like Dr Frankenstein, oblivious to their own folly.
  8. A system of sustainable agriculture, with long time horizons. In this system, short term improvements in agricultural yield are largely irrelevant, unless the improvement can be demonstrated to be sustainable indefinitely without further substantial inputs to the system. In general, most technological fixes fail this test. The green revolution was brought about by a massive reliance on artificial fertilizer, derived from fossil fuels. As we hit peak oil, this approach cannot be sustained. Additionally, the approach has brought its own problems, including a massive nitrogen pollution of lakes and coastal waters, and poorer quality soils, and of course, the resulting climate change from the heavy use of fossil fuels. In this sense, technological fixes provide short term gains in exchange for a long term debt that must be paid by future generations. In this view, GMOs are seen as an even bigger step in the wrong direction, as they replace an existing diversity in seed-stocks and farming methods with industrialized mono-cultures, and divert attention away from the need for soil conservation, and long-term sustainable farming practices. In this system, small scale organic farming is seen as the best way of improving the resilience of the global food production. While organic farming sometime (but not always!) means lower yields, it reduces dependency on external inputs (e.g. artificial fertilizers and pesticides), and increases diversity. Systems with more diverse structures tend to be more resilient in the face of new threats, and the changing climates over the next few decades will severely test the resilience of our farming methods in many regions of the world.  The people who worry about this system point to failures of GMOs to maintain their resistance to pests. Here, you get a reinforcing feedback loop in which you need ever more advances in GMO technology to keep pace with the growth of resistance within the ecosystem, and with each such advance, you make it harder for non-GMO food varieties to survive. So while most proponents of GMOs see them as technological saviours, in the long term it’s likely they actually reduce the ability of the global agricultural system to survive the shocks of climate change.

Systems theory leads us to expect that these systems will interact in interesting ways, and indeed they do. For example, systems 6 and 8 can easily be confused as having the same goal, but in fact, because the systems have very different temporal scales, they can end up being in conflict: short-term improvements to agricultural yield can lead to long term reduction of sustainability and resilience. Systems 6 and 7 can also interfere – it’s been argued that the green revolution reduced world starvation and replaced it with widespread malnutrition, as industrialization of food production gives us fewer healthy food choices. Systems 1 and 4 are often in conflict, and are leading to ever more heated debates over open access to research results. And of course, one of the biggest worries of some of the protest groups is the interaction between systems 2 and 5: the existence of a large profit motive tends to weaken good risk management practices in biotech research.

Perhaps the most telling interaction is the opportunity cost. While governments and corporations, focusing on systems 5 & 6, pour funding and effort into research into GMOs, other, better solutions to long term sustainability and resilience, required in system 8, become under-invested. More simply: if we’re asking the wrong question about the benefit of GMOs, we’ll make poor decisions about whether to pursue them. We should be asking different questions about how to feed the world, and resources put into publicly funded GMO research tend to push us even further in the wrong direction.

So where does that leave the proposed protests? Should the trials at Rothamsted be allowed to continue, or do the protesters have the right to force an end to the experiment, by wilful destruction if necessary? My personal take is that the experiment should be halted immediately, preferably by Rothamsted itself, on the basis that it hasn’t yet passed the test for beneficence in a number of systems. The knowledge gain from this one trial is too small to justify creating this level of societal conflict. I’m sure some of my colleague will label me anti-science for this position, but in fact, I would argue that my position here is strongly pro-science: an act of humility by scientists is far more likely to improve the level of trust that the public has in the scientific community. Proceeding with the trial puts public trust in scientists further at risk.

Let’s return to that question of whether there’s an analogy between people attacking the biotech scientists and people attacking climate scientists. If you operate purely within system 1, the analogy seems compelling. However, it breaks down as soon as you move to system 2, because the risks have opposite signs. In the case of GMO food trials, the research itself creates a risk; choosing not to do the research at all (or destroying it if someone else tries it) is an attempt to reduce risk. In the case of climate science, the biggest risks are on the business-as-usual scenario. Choosing to do the research itself poses no additional risk, and indeed reduces it, because we come to understand more about how the climate system works.

The closest analogy in climate science I can think of is the debate over geo-engineering. Many climate scientists objected to any research being done on geo-engineering for many years, for exactly the reason many people object to GMO research – because it diverts attention away from more important things we should be doing, such as reducing greenhouse gas emissions. A few years back, the climate science community seems to have shifted perspective, towards the view that geo-engineering is a desperate measure that might buy us more time  to get emissions under control, and hence research is necessary to find out how well it works. A few geo-engineering field trials have already happened. As these start to gain more public attention, I would expect the protests to start in earnest, along with threats to destroy the research. And it will be for all the same reasons that people want to destroy the GM wheat trials at Rothamsted. And, unless we all become better systems thinkers, we’ll have all the same misunderstandings.

Update (May 29, 2012): I ought to collect links to thought provoking articles on this. Here are some:

There’s a fascinating piece up this week on The Grid on how to make Toronto a better city. They asked a whole bunch of prominent people for ideas, each to be no longer than 200 words. The ideas didn’t necessarily have to be practical, but would be things to make us think. Some of them are wacky, some are brilliant, and some are both. My favourites are:

  • Give people alternative ways to pay their dues, e.g. instead of taxes, struggling artists donate public art, etc. (Seema Jethalal);
  • Hold a blackout holiday twice a year, to mimic the sense of connectness we all got when the power grid went down in 2003 (Carlyle Jansen)
  • Use ranked ballots for all municipal elections (Dave Meslin)
  • Banish all outdoor commercial ads (Sean Martindale)
  • Ban parking on all main streets (Chris Hume)
  • Build a free wireless internet via decentralized network sharing (Jesse Hirsh)
  • Make the TTC (our public transit) free (David Mirvish)

Better yet, they asked for more suggestions from readers. Here are mine:

Safe bike routes to schools. Every school should be connected to a network of safe bike paths for kids. Unlike the city’s current bike network, these bike baths should avoid main roads as much as possible: bike lanes on main roads are not safe for kids. Instead they should go via residential streets, parks, and marginal spaces, and physically separate the bikes from all vehicle traffic. These routes should provide uninterrupted links from sheltered bike parking at each school all the way through the  residential neighbourhoods that each school serves. They should also provide a larger network, linking each school with neighbouring schools, for families where the kids go to different local schools, and where kids use services (e.g. pools) in other local schools.

Advantages: kids get exercise biking to school, gain some independence from parents, and become better connected with their environment. Traffic congestion and pollution at school drop-off and pickup times would drop. To build such a network, we would have to sacrifice some on-street parking in residential streets. However, a complete network of such bike paths could become a safer alternative to the current bike lanes on main streets, thus freeing up space on main streets.


Car-free blocks on streetcar routes. On each streetcar route through the city, select individual blocks (i.e. stretches between adjacent cross-streets) at several points along each route, and close these stretches to all other motorized vehicle traffic. Such blocks would only allow pedestrians, bikes and streetcars. The sidewalks would then be extended for use as patios by cafes and restaurants. Delivery vehicles would still be permitted, perhaps only at certain times of day.

The aim is to discourage other traffic from using the streets that our streetcars run on as major commuting corridors through the city, and to speed up the flow of streetcars. The blocks selected to pedestrianize would be those where there is already a lively street life, with existing cafes, etc. Such blocks would become desirable destinations for shoppers, diners and tourists.

I’ve been working for the past couple of months with the Cities Centre and the U of T Sustainability Office to put together a symposium on sustainability, where we pose the question “What role should the University of Toronto play in the broader challenge of building a sustainable city?”. We now finally have all the details in place:

  • An Evening Lecture, on the evening of June 13, 6pm to 9pm, at Innis Town Hall, featuring Bob Willard, author of “The Sustainability Advantage”, Tanzeel Merchant, of the Ontario Growth Secretariat and Heritage Toronto, and John Robinson, Director of the UBC Sustainability initiative and the Royal Canadian Geographical Society’s Canadian Environmental Scientist of the Year.
  • A full day visioning workshop on June 14, 9am to 5pm, at the Debates Room, Hart House. With a mix of speakers and working group sessions, the goal will be to map out a vision for sustainability at U of T, that brings together research, teaching and operations at the University, and explores how we can use the University as a “living lab” to investigate challenges in urban sustainability.

And it’s free. Register here!

On my trip to Queens University last week, I participated in a panel session on the role of social media in research. I pointed out that tools like twitter provide a natural extension to the kinds of conversations we usually only get to have at conferences – the casual interactions with other researchers that sometimes lead to new research questions and collaborations.

So, with a little help from Storify, here’s an example…

In which we see and example of how Twitter can enable interesting science, and understand a little about the role of existing social networks in getting science done.


At the CMIP5 workshop earlier this week, one of Ed Hawkins‘ charts caught my eye, because he changed how we look at model runs. We’re used to seeing climate models used to explore the range of likely global temperature responses under different future emissions scenarios, and the results presented as a graph of changing temperature over time. For example, this iconic figure from the last IPCC assessment report (click for the original figure and caption at the IPCC site):

These graphs tend to focus too much on the mean temperature response in each scenario (where ‘mean’ means ‘the multi-model mean’). I tend to think the variance is more interesting – both within each scenario (showing differences in the various CMIP3 models on the same scenarios), and across the different scenarios (showing how our future is likely to be affected by the energy choices implicit in each scenario). A few months ago, I blogged about the analysis that Hawkins and Sutton did on these variabilities, to explore how the different sources of uncertainty change as you move from near term to long term. The analysis shows that in the first few decades, the differences in the models dominate (which doesn’t bode well for decadal forecasting – the models are all over the place). But by the end of the century, the differences between the emissions scenarios dominates (i.e. the spread of projections from the different scenarios is significantly bigger than the  disagreements between models). Ed presented an update on this analysis for the CMIP5 models this week, which looks very similar.

But here’s the new thing that caught my eye: Ed included a graph of temperature responses tipped on its side, to answer a different question: how soon will the global temperature exceed the policymaker’s adopted “dangerous” threshold of 2°C, under each emissions scenario. And, again, how big is the uncertainty? This idea was used in a paper last year by Joshi et. al., entitled Projections of when temperature change will exceed 2 °C above pre-industrial levels. Here’s their figure 1:

Figure 1 from Joshi et al, 2011

By putting the dates on the Y-axis and temperatures on the X-axis, and cutting off the graph at 2°C, we get a whole new perspective on what the models runs are telling us. For example, it’s now easy to see that in all these scenarios, we pass the 2°C threshold well before the end of the century (whereas the IPCC graph above completely obscures this point), and under the higher emissions scenarios, we get to 3°C by the end of the century.

A wonderful example of how much difference the choice of presentation makes. I guess I should mention, however, that the idea of a 2°C threshold is completely arbitrary. I’ve asked many different scientists where the idea came from, and they all suggest it’s something the policymakers dreamt up, rather than anything arising out of scientific analysis. The full story is available in Randalls, 2011, “History of the 2°C climate target”.

In the talk I gave this week at the workshop on the CMIP5 experiments, I argued that we should do a better job of explaining how climate science works, especially the day-to-day business of working with models and data. I think we have a widespread problem that people outside of climate science have the wrong mental models about what a climate scientist does. As with any science, the day-to-day work might appear to be chaotic, with scientists dealing with the daily frustrations of working with large, messy datasets, having instruments and models not work the way they’re supposed to, and of course, the occasional mistake that you only discover after months of work. This doesn’t map onto the mental model that many non-scientists have of “how science should be done”, because the view presented in school, and in the media, is that science is about nicely packaged facts. In reality, it’s a messy process of frustrations, dead-end paths, and incremental progress exploring the available evidence.

Some climate scientists I’ve chatted to are nervous about exposing more of this messy day-to-day work. They already feel under constant attack, and they feel that allowing the public to peer under the lid (or if you prefer, to see inside the sausage factory) will only diminish people’s respect for the science. I take the opposite view – the more we present the science as a set of nicely polished results, the more potential there is for the credibility of the science to be undermined when people do manage to peek under the lid (e.g. by publishing internal emails). I think it’s vitally important that we work to clear away some of the incorrect mental models people have of how science is (or should be) done, and give people a better appreciation for how our confidence in scientific results slowly emerges from a slow, messy, collaborative process.

Giving people a better appreciation of how science is done would also help to overcome some of games of ping pong you get in the media, where each new result in a published paper is presented as a startling new discovery, overturning previous research, and (if you’re in the business of selling newspapers, preferably) overturning an entire field. In fact, it’s normal for new published results to turn out to be wrong, and most of the interesting work in science is in reconciling apparently contradictory findings.

The problem is that these incorrect mental models of how science is done are often well entrenched, and the best that we can do is to try to chip away at them, by explaining at every opportunity what scientists actually do. For example, here’s a mental model I’ve encountered from time to time about how climate scientists build models to address the kinds of questions policymakers ask about the need for different kinds of climate policy:

This view suggests that scientists respond to a specific policy question by designing and building software models (preferably testing that the model satisfies its specification), and then running the model to answer the question. This is not the only (or even the most common?) layperson’s view of climate modelling, but the point is that there are many incorrect mental models of how climate models are developed and used, and one of the things we should strive to do is to work towards dislodging some of these by doing a better job of explaining the process.

With respect to climate model development, I’ve written before about how models slowly advance based on a process that roughly mimics the traditional view of “the scientific method” (I should acknowledge, for all the philosophy of science buffs, that there really isn’t a single, “correct” scientific method, but let’s keep that discussion for another day). So here’s how I characterize the day to day work of developing a model:

Most of the effort is spent identifying and diagnosing where the weaknesses in the current model are, and looking for ways to improve them. Each possible improvement then becomes an experiment, in which the experimental hypothesis might look like:

“if I change <piece of code> in <routine>, I expect it to have <specific impact on model error> in <output variable> by <expected margin> because of <tentative theory about climactic processes and how they’re represented in the model>”

The previous version of the model acts as a control, and the modified model is the experimental condition.

But of course, this process isn’t just a random walk – it’s guided at the next level up by a number of influences, because the broader climate science community (and to some extent the meteorological community) are doing all sorts of related research, which then influences model development. In the paper we wrote about the software development processes at the UK Met Office, we portrayed it like this:

But I could go even broader and place this within a context in which a number of longer term observational campaigns (“process studies”) are collecting new types of observational data to investigate climate processes that are still poorly understood. This then involves the interaction several distinct communities. Christian Jakob portrays it like this:

Although the point of Jakob’s paper is to argue that the modelling and process studies communities don’t currently do enough of this kind of interactions, so there’s room for improvement in how the modelling influences the kinds of process studies needed, and how the results from process studies feed back into model development.

So, how else should we be explaining the day-to-day work of climate scientists?