31. May 2010 · 5 comments · Categories: psychology

I’m fascinated by the cognitive biases that affect people’s perceptions of climate change. I’ve previously written about the Dunning-Kruger effect (the least competent people tend to vastly over-rate their competence), and Kahan and Braman’s studies on social epistemology (people tend to ignore empirical evidence if its conclusions contradict their existing worldview).

Now comes a study by Nyhan and Reifler in the journal Political Behaviour entitled “When Corrections Fail: The persistence of Political Misperceptions“. N&R point out that in the literature, there is plenty of evidence that people often “base their policy preferences on false, misleading or unsubstantiated information that they believe to be true”. Studies have shown that providing people with correct factual information doesn’t necessarily affect their beliefs. However, different studies disagree about the latter point, partly because it’s often not clear in these studies whether the subject’s beliefs changed at all, and partly because previous studies have differed over how the factual information is presented (and even what counts as ‘factual’).

So N&R set out to directly study whether corrective information does indeed change erroneous beliefs. Most importantly, they were interested in what happens when this corrective information isn’t presented directly as an authoritative account of the truth, but rather (as happens more often) when it is presented as part of a larger, more equivocal set of stories in the media. One obvious factor that causes people to preserve erroneous beliefs is through selective reading – people tend to seek out information that supports their existing beliefs; hence they often don’t encounter information that corrects their misperceptions. And even when people do encounter corrective information, they are more likely to reject it (e.g. by thinking up counter-arguments) if it contradicts their prior beliefs. It is this latter process that N&R investigated, and in particular, whether this process of thinking up counter-arguments can actually reinforce the misperception; they dub this a “correction backfire”.

Four studies were conducted. In each case, the subjects were presented with a report of a speech by a well-known public figure. When a factual correction was added to the article, those subjects who are most likely to agree with the contents of the speech were unmoved by the factual correction, and in several of the studies, the correction actually strengthened their belief in the erroneous information (i.e. a clear ‘correction backfire’):

  • Study 1 conducted in the fall of 2005, examined beliefs in whether Iraq had weapons of mass destruction prior to the US invasion. Subjects were presented with a newspaper article describing a speech by president Bush in which he talks about the risk of Iraq passing these weapons on to terrorist networks. In the correction treatment, the article goes on to describe the results of the Duelfer report, which concluded there were, in fact, no weapons of mass destruction. The result shows a clear correction backfire for conservatives – the correction significantly increased their belief that Iraq really did have such weapons, while for liberals, the correction clearly decreased their belief.
  • Study 2, conducted in the spring of 2006 repeats study 1, with some variation in the wording. This study again showed that the correction was ineffective for conservatives – it didn’t decrease their belief in the existence of the weapons. However, unlike study 1, it didn’t show a correction backfire, although a re-analysis of the results indicated that there was such a backfire among those conservatives who most strongly supported the Iraq war. This study also attempted to test the effect of the source of the newspaper report – i.e. does it matter if it’s presented as being from the New York Times (perceived by many conservatives to have a liberal bias) or Fox News (perceived as being conservative)? In this case, the source of the article made no significant difference.
  • Study 3, also conducted in 2006, examined the belief that the Bush tax cuts paid for themselves by stimulating enough economic growth to actually increase government revenues. Subjects were presented with an article in which president Bush indicated the tax cuts had helped to increase revenues. In the correction treatment, the article goes on to present the actual revenues, showing that tax revenues declined sharply (both absolutely, and as a proportion of GDP) in the years after the tax cuts were enacted. Again there was a clear correction backfire among conservatives – those receiving the article presenting the actual revenues actually increased their belief that the tax cuts paid for themselves.
  • Study 4, also from 2006, examined the belief that Bush banned stem cell research. Subjects were presented with an article describing speeches by senators Edwards and Kerry in which they suggest such a ban exists. In the corrective treatment, a paragraph was added to explain that Bush didn’t actually ban stem cell research, because his restrictions didn’t apply to privately funded research. The results were that the correction did not change liberal’s belief that there was such a ban, but there was no correction backfire (i.e. it didn’t increase their beliefs in the ban).

In summary, factual corrections in newspaper articles don’t appear to work for those who are ideologically motivated to hold the misperception, and in two out of the four studies, it actually strengthened the misperception. So, fact-checking on its own is not enough to overcome ideologically-driven beliefs. (h/t to Ben Goldacre for this)

How does this relate to climate change? Well, most media reports on climate change don’t even attempt any fact-checking anyway – they ignore the vast body of assessment reports by authoritative scientific bodies, and present a “he-said-she-said” slugfest between denialists and climate scientists. The sad thing is that the addition of fact-checking won’t, on it’s own, make any difference to those whose denial of climate change is driven by their ideological leanings. If anything, such fact-checking will make them even more entrenched…

Dear God, I would like to file a bug report

(clickety-click for the full xkcd cartoon)

I’ve been working with group of enthusiastic parents in our neighbourhood over the past year on a plan to make our local elementary school a prototype for low-energy buildings. As our discussions evolved, we ended up with a much more ambitious vision: to use the building and grounds of the school for renewable power generation projects (using solar, and geothermal energy) that could potentially power many of the neighbouring houses and condos – i.e. make the school a community energy hub. And of course, engage the kids in the whole process, so that they learn about climate and energy, even as we attempt to build solutions.

In parallel with our discussions, the school board has been beefing up its ambitions too, and has recently adopted a new Climate Change Action Plan. It makes for very interesting reading. I like the triple goal: mitigation, adaptation and education, largely because the last of these, education, is often missing from discussions about how to respond to climate change, and I firmly believe that the other two goals depend on it. The body of the report is a set of ten proposed actions to cut carbon emissions from the buildings and transportation operated by the school board, funded from a variety of sources (government grants, the feed-in tariff program, operational savings, carbon credits, etc). The report still needs some beefing up on the education front, but it’s a great start!

Here are two upcoming conferences, both relevant to the overlap of computer science and climate science:

…and I won’t make it to either as I’ll be doing my stuff at NCAR. I will get to attend this though:

I guess I’ll have to send some of my grad students off to the other conferences (hint, hint).

I’ve been busy the last few weeks setting up the travel details for my sabbatical. My plan is to visit three different climate modeling centers, to do a comparative study of their software practices. The goal is to understand how the software engineering culture and practices vary across different centers, and how the differences affect the quality and flexibility of the models. The three centers I’ll be visiting are:

I’ll spend 4 weeks at each centre, starting in July, running through to October, after which I’ll spend some time analyzing the data and writing up my observations. Here’s my research plan…

Our previous studies at the UK Met Office Hadley Center suggest that there are many features of software development for earth system modeling that make it markedly different from other types of software development, and which therefore affect the applicability of standard software engineering tools and techniques. Tools developed for commercial software tend not to cater for the demands of working with high performance code for parallel architectures, and usually do not fit well with the working practices of scientific teams. Scientific code development has challenges that don’t apply to other forms of software: the need to keep track of exactly which version of the program code was used in a particular experiment, the need to re-run experiments with precisely repeatable results, the need to build alternative versions of the software from a common code base for different kinds of experiments. Checking software “correctness” is hard because frequently the software must calculate approximate solutions to numerical problems for which there is no analytical solution. Because the overall goal is to build code to explore a theory, there is no oracle for what the outputs should be, and therefore conventional approaches to testing (and perhaps code quality in general) don’t apply.

Despite this potential mismatch, the earth system modeling community has adopted (and sometimes adapted) many tools and practices from mainstream software engineering. These include version control, bug tracking, automated build and test processes, release planning, code reviews, frequent regression testing, and so on. Such tools may offer a number of potential benefits:

  • they may increase productivity by speeding up the development cycle, so that scientists can get their ideas into working code much faster;
  • they may improve verification, for example using code analysis tools to identify and remove (or even prevent) software errors;
  • they may improve the understandability and modifiability of computational models (making it easier to continue to evolve the models);
  • they may improve coordination, allowing a broader community to contribute to and make use of a shared the code base for a wider variety of experiments;
  • they may improve scalability and performance, allowing code to be configured and optimized for a wider variety of high performance architectures (including massively parallel machines), and for a wider variety of grid resolutions.

This study will investigate which tools and practices have been adopted at the different centers, identify differences and similarities in how they are applied, and, as far as is possible, assess the effectiveness of these practices. We will also attempt to characterize the remaining challenges, and identify opportunities where additional tools and techniques might be adopted.

Specific questions for the study include:

  1. Verification – What techniques are used to ensure that the code matches the scientists’ understanding of what it should do? In traditional software engineering, this is usually taken to be a question of correctness (does the code do what it is supposed to?); however, for exploratory modeling it is just as often a question of understanding (have we adequately understood what happens when the model runs?). We will investigate the practices used to test the code, to validate it against observational data, and to compare different model runs against one another, and assess how effective these are at eliminating errors of correctness and errors of understanding.
  2. Coordination – How are the contributions from across the modeling community coordinated? In particular, we will examine the challenges of synchronizing the development processes for coupled models with the development processes of their component models, and how the differences in the priorities of different, overlapping communities of users affect this coordination.
  3. Division of responsibility – How are the responsibilities for coding, verification, and coordination distributed between different roles in the organization? In particular, we will examine how these responsibilities are divided across the scientists and other support roles such as ‘systems’ or ‘software engineering’ personnel. We will also explore expectations on the quality of contributed code from end-user scientists, and the potential for testing and review practices to affect the quality of contributed code.
  4. Planning and release processes – How do modelers decide on priorities for model development, how do they decide which changes to tackle in a particular release of the model, and how they navigate between computational feasibility and scientific priorities? We will also investigate how the change process is organized, how changes are propagated to different sub-communities.
  5. Debugging – How do scientists currently debug the models, what types of bugs do they find in their code currently, and how they find them? In particular, we will develop a categorization of model errors, to use as a basis for subsequent studies into new techniques for detecting and/or eliminating such errors.

The study will be conducted through a mix of interviews and observational studies, focusing on particular changes to the model codes developed at each center. The proposed methodology is to identify a number of candidate code changes, including recently completed changes and current work-in-progress, and to build a “life story” for each such change, covering how each change was planned and conducted, what techniques were applied, and what problems were encountered. This will lead to a more detailed description of the current software development practices, which can then be compared and contrasted with studies of practices used for other types of software. This end result will be an identification of opportunities where existing tools and techniques can be readily adapted (with some clear indication of the potential benefits), along with a longer-term research agenda for problem areas where no suitable solutions currently exist.

This week we’re demoing Inflo at the Ontario Centres of Excellence Discovery Conference 2010. It’s given me a chance to play a little more with the demo, and create some new sample calculations (with Jonathan valiantly adding new features on the fly in response to my requests!). The idea of Inflo is that it should be an open source calculation tool – one that supports a larger community of people discussing and reaching consensus on the best way to calculate the answer to some (quantifiable) question.

For the demo this week, I re-did the calculation on how much of the remaining global fossil fuel reserves we can burn and still keep global warming within the target threshold of a +2°C rise over pre-industrial levels. I first did this calculation in blog post back in the fall, but I’ve been keen to see if Inflo would provide a better way of sharing the calculation. Creating the model is still a little clunky (it is, after all, a very preliminary prototype), but I’m pleased with the results. Here’s a screenshot:

And here’s a live link to try it out. A few tips: the little grey circles under a node indicate there’s some hidden subtrees. Double-clicking on one of these will expand it, while double clicking on an expanded node will collapse everything below it, so you can explore the basis for each step in the calculation. The Node Editor tool bar on the left shows you the formula for the selected node, and any notes. Some of the comments in the “Description” field are hotlinks to data sources – mouseover the text to find them. Oh, and the arrows don’t always update properly when you change views – selecting a node in the graph should force them to update. Oh, and the units are propagated (and scaled for readability) automatically, which is why they sometime look a little odd, eg. “tonne of carbon” rather than “tonnes”. One of our key design decisions is to make the numbers as human-readable as possible, and always ensure correct units are displayed.

The demo should get across some of what we’re trying to do. The idea is to create a visual, web-based calculator that can be edited and shared; eventually we hope to build wikipedia-like communities who will curate the calculations, to ensure that the appropriate sources of data are used, and that the results can be trusted. We’ll need to add more facilities for version management of calculations, and for linking discussions to (portions of) the graphs.

Here’s another example: Jono’s carbon footprint analysis of whether you should print a document or read it on the screen (double click the top node to expand the calculation).

03. May 2010 · 1 comment · Categories: ICSE 2010

Today is our second workshop on software research and climate change, at ICSE 2010 in Cape Town. We’ve finalized the program, and we’re hoping to support some form of remote participation, but I’m still not sure how this will work out.

We had sixteen position papers and two videos submitted in the end, which I’m delighted about. To get everyone reading and discussing them prior to the workshop, we set up an open reviewing process, which I think went very well. Rather than the usual closed, anonymous reviews, we opened it up so that everyone could add reviews to any paper, and we encouraged everyone to review in their own name, rather than anonymously. The main problem we had was finding a suitable way of supporting this – until we hit upon the idea of creating a workshop blog, so each paper is a blog post, and the comment thread allows us to add reviews, and comment on each other’s reviews. This is nice because it means we can now make all the papers and reviews public, and continue the discussions during and after the workshop.

We’re trying out two different ways of supporting live remote participation – in the morning, the keynote talk (by Stephen Emmott of Microsoft Research) will be delivered via Microsoft’s LiveMeeting. We tested it out last week, and I’m pretty impressed with it (apart from the fact that there’s no client for the Mac). The setup we’ll be using is to have a video feed of Stephen giving the talk, displayed on a laptop screen at the front of the room, with his slides projected to the big screen. The laptop also has a webcam, so (if it works) Stephen will be able to see his audience too. I’ll document how well this works in a subsequent post.

For the last afternoon session, we’ll be trying out a live skype call. Feel free to send me your skype details if you’d like to participate. I’ve no idea if this will work (as it didn’t last time we tried), but hey, it’s worth exploring…