Applying systems thinking to computing, climate and sustainability

Andrew Wisdom – Cities as Systems of Systems

28. October 2012 · 3 comments · Categories: sustainability, systems thinking

The second speaker at our Workshop on City Science was Andrew Wisdom from Arup, talking about Cities as Systems of Systems. Andrew began with the observation that cities are increasingly under pressure, as the urban population continues to grow, and cities struggle to provide adequate infrastructure for their populations to thrive. But a central part of his message is that the way we think about things tends to create the way they are, and this is especially so with how we think about our cities.

As an exercise, he first presented a continuum of worldviews, from Technocentric at one end, to Ecocentric at the other end:

In the Techno-centric view, humans are dissociated from the earth. Nature has no inherent value, and we can solve everything with ingenuity and technology. This worldview tends to view the earth as an inert machine to be exploited.
In the Eco-centric view, the earth is alive and central to the web of life. Humans are an intrinsic part of nature, but human activity is already exceeding the limits of what the planet can support, to the point that environmental problems are potentially catastrophic. Hence, we need to get rid of materialism, eliminate growth, and work to restore balance.
Somewhere in the middle is a Sustain-centric view, which accepts that the earth provides an essential life support system, and that nature has some intrinsic value. This view accepts that limits are being reached, that environmental problems tend to take decades to solve, and that more growth is not automatically good. Humans can replace some but not all natural processes, and we have to focus more on quality of life as a measure of success.

As an exercise, Andrew asked the audience to imagine this continuum spread along one wall of the room, and asked us each to go and stand where we felt we fit on the spectrum. Many of the workshop participants positioned themselves somewhere between the eco-centric and sustain-centric views, with a small cluster at the extreme eco-centric end, and another cluster just to the techno-centric side of sustain-centric. Nobody stood at the extreme techno-centric end of the room!

Then, he asked us to move to where we think the city of Toronto sits, and then where we think Canada sits, and finally where we feel the world sits. For the first two of these, everyone shifted a long way towards the technocentric end of the spectrum (and some discussion ensued to the effect that both our mayor and our prime minister are a long way off the chart altogether – they are both well known for strong anti-environmentalist views). For the whole world, people didn’t move much from the “Canada” perspective. An immediate insight was that we (workshop attendees) are far more towards the ecocentric end of the spectrum than either our current city or federal governments, and perhaps the world in general. So if our governments (and by extension the voters who elect them) are out of step with our own worldviews, what are the implications? Should we, as researchers, be aiming to shift people’s perspectives?

One problem that arises from one’s worldview is how people understand messages about environmental problems. For example, people with a technocentric perspective tend to view discussions of sustainability as being about sacrifice – ‘wearing a hair shirt’, consume less, etc. Which then leads to a waning interest in these topics. For example, analysis of google trends on terms like global warming and climate change show spikes in 2007 around the release of Al Gore’s movie and the IPCC assessment, but declining interest since then.

Jeb Brugmann, the previous speaker, talked about the idea of a Consumptive city versus a Generative city, which is a change in perspective that alters how we view cities, changes what we choose to measure, and hence affects the way our cities evolve.

Changes in the indices we pay attention to can have a dramatic impact. For example, a study in Melbourne created that VAMPIRE index (Vulnerability Assessment for Mortgage, Petroleum and Inflation Risks and Expenses), which shows the relative degree of socio-economic stress in suburbs in Brisbane, Sydney, Melbourne, Adelaide and Perth. The pattern that emerges is that in the Western suburbs of Melbourne, there are few jobs, and many people paying off mortgages, all having to commute and hour and a half to the east of the city for work.

Our view of a city tend to create structures that compartmentalize different systems into silos, and then we attempt to optimize within these silos. For example, zoning laws create chunks of land with particular prescribed purposes, and then we end up trying to optimize within each zone. When zoning laws create the kind of problem indicated by the Melbourne VAMPIRE index, there’s little the city can do about it if they continue to think in terms of zoning. The structure of these silos has become fossilized into the organizational structure of government. Take transport, for example. We tend to look at existing roads, and ask how to widen them to handle growth in traffic; we rarely attempt to solve traffic issues by asking bigger questions about why people choose to drive. Hence, we miss the opportunity to solve traffic problems by changing the relationship between where people live and where they work. Re-designing a city to provide more employment opportunities in neighbourhoods that are suffering socio-economic stress is far more likely to help than improving the transport corridors between those neighbourhoods and other parts of the city.

Healthcare is another example. The outcome metrics typically used for hospital use include average length of stay, 30-day unplanned readmission rate, cost of readmission, etc. Again, these metrics create a narrow view of the system – a silo – that we then try to optimize within. However, if you compare European and American healthcare systems, there are major structural difference. The US system is based on formula funding, in which ‘clients’ are classified in terms of type of illness, standard interventions for that illness, and associated costs. Funding is then allocated to service providers based on this classification scheme. In Europe, service provides are funded directly, and are able to decide at the local level how best to allocate that funding to serve the needs of the population they care for. The European model is a much more flexible system that treats patients real needs, rather than trying to fit each patient into a pre-defined category. In the US, the medical catalogue of disorders becomes an accounting scheme for allocating funds, and the result is that in the US, medical care costs going up faster than any other country. If you plot life expectancy against health spending, the US is falling far behind:

The problem is that the US health system views illness as a problem to be solved. If you think in terms of wellbeing rather than illness, you broaden the set of approaches you can use. For example, there are significant health benefits to pet ownership, providing green space within cities, and so on, but these are not fundable with the US system. There are obvious connections between body mass index and the availability of healthy foods, the walkability of neighbourhoods, and so on, but these don’t fit into a healthcare paradigm that allocates resources according to disease diagnosis.

Andrew then illustrated the power of re-thinking cities as systems-of-systems through several Arup case studies:

Dongtan eco-city. This city was designed from the ground up to be food positive, and energy positive (ie. intended to generate more food and more clean energy than it uses). The design makes it more preferable to walk or bike than to drive a car. A key design tool was the use of an integrated model that captures the interactions of different systems within the city. [Dongtan is, incidentally, a classic example of how the media alternately overhypes and then trashtalks major sustainability initiatives, when the real story is so much more interesting].
Low2No, Helsinki, a more modest project that aims to work within the existing city to create carbon negative buildings and energy efficient neighbourhoods step by step.
Werribee, a suburb of Melbourne, which is mainly an agricultural town, particularly known for its broccoli farming. But with fluctuating prices, farmers have had difficulty selling their broccoli. In an innovative solution that turns this problem into an opportunity, Arup developed a new vision that uses local renewable energy, water and waste re-processing to build a self-sufficient hothouse food production and research facility that provides employment and education along with food and energy.

In conclusion, we have to understand how our views of these systems constrain us to particular pathways, and we have to understand the connections between multiple systems if we want to understand the important issues. In many cases, we don’t do well at recognizing good outcomes, because our worldviews lead us to the wrong measures of success, and then we use these measures to create silos, attempting to optimize within them, rather than seeing the big picture. Understanding the systems, and understanding how these systems shape our thinking is crucial. However, the real challenges then lie in using this understanding to frame effective policy and create effective action.

After Andrew’s talk, we moved into a hands-on workshop activity, using a set of cards developed by Arup called Drivers of Change. The cards are fascinating – there are 189 cards in the deck, each of which summarizes a key issue (e.g. urban migration, homelessness, clean water, climate change, etc), and on the back, distills some key facts and figures. Our exercise was to find connections between the cards – each person had to pick one card that interested him or her, and then team up with two other people to identify how their three cards are related. It was a fascinating and thought-provoking exercise, that really got us thinking about systems-of-systems. I’m now a big fan of the cards and plan to use them in the classroom. (I bought a deck at Indigo for $45, although I note that, bizarrely, Amazon has them selling for over $1000!).

Jeb Brugmann – The Productive City

17. October 2012 · Write a comment · Categories: books, sustainability

We held a 2-day workshop at U of T last week entitled “Finding Connections – Towards a Holistic View of City Systems“. The workshop brought together a multi-disciplinary group of people from academia, industry, government, and the non-profit sector, all of whom share a common interest in understanding how cities work as systems-of-systems, and how to make our cities more sustainable and more liveable. A key theme throughout the workshop was how to make sure the kinds of research we do in universities does actually end up being useful to decision-makers – i.e. can we strengthen evidence-based policymaking (and avoid, as one of the participants phrased it, “policy-based evidence-making”).

I plan to blog some of the highlights of the workshop, starting with the first keynote speaker.

The workshop kicked off with an inspiring talk by Jeb Brugmann, entitled “The Productive City”. Jeb is an expert in urban sustainability and climate change mitigation, and has a book out called “Welcome to the Urban Revolution: How Cities are Changing the World“. (I should admit the book’s been sitting on the ‘to read’ pile on my desk for a while – now I have to read it!).

Jeb’s central message was that we need to look at cities and sustainability in a radically different way. Instead of thinking of sustainability as about saving energy, living more frugally, and making sacrifices, we should be looking out how we re-invent cities as places that produce resources rather than consume them. And he offered a number of case studies that demonstrate how this is already possible.

Jeb started his talk with the question: How will 9 billion people thrive on Earth? He then took us back to a UN meeting in 1990, the World Congress of Local Governments for a Sustainable Future. This meeting was the first time that city governments around the world came together to grapple with the question of sustainable development. To emphasis how new this was, Jeb recollected lengthy discussions at the meeting on basic questions such as how to translate the term ‘sustainable development’ into French, German, etc.

The meeting had two main outcomes:

Initial work on Agenda 21, getting communities engaged in collaborative sustainable decision making. [Note: Agenda 21 was subsequently adopted by 178 countries at the Rio Summit in 1992. More interestingly, if you google for Agenda 21 these days, you’re likely to find a whole bunch of nutball right-wing conspiracy theories about it being an agenda to destroy American freedom.]
A network of city governments dedicated to developing action on climate change [This network became ICLEI – Local Governments for Sustainability]. Jeb noted how the ambitions of the cities participating in ICLEI have grown over the years. Initially, many of these cities set targets around 20% reduction in greenhouse gas emissions. Over the years since, these target have grown. For example, Chicago now has a target of 80% reduction. This is significant because these targets have been through city councils, and have been discussed and agreed on by those councils.

An important idea arising out of these agreements is the concept of the ecological footprint – sometimes expressed as how many earths are needed to support us if everyone had the same resource consumption as you. The problem is that you get into definitional twists on how you measure this, and that gets in the way of actually using it as a productive planning tool.

Here’s another way of thinking about the problem. Cities currently have hugely under-optimized development patterns. For example, cities with seven times more outspill growth (suburban sprawl) compared to infill growth. But there are emergent pressures on industry to optimize use of urban space and urban geography. Hence, we should start to examine under-used urban assets. If we can identify space within the city that doesn’t generate value, we can reinvent it. For example, the laneways of Melbourne, which in the 1970’s and 80’s were derelict, have now been regenerated for a rich network of local stores and businesses, and ended up as a major tourist attraction.

We also tend to dramatically underestimate the market viability of energy efficient, sustainable buildings. For example, in Hannover, a successful project built an entire division of eco-homes using Passivhaus standards at similar rental price to the old 1960s apartment buildings.

The standard view of cities, built into the notion of ecological footprint, is that cities are extraction engines – the city acts as a machine that extracts resources from the surrounding environment, processes these resources to generate value, and produces waste products that must be disposed of. Most work on sustainable cities frames the task as an attempt to reduce the impact of this process, by designing eco-efficient cities. For example, the use of secondary production (e.g. recycling) and designed dematerialization (reduction of waste in the entire product lifecycle) to reduce the inflow of resources and the outflow of wastes.

Jeb argues a more audacious goal is needed: We should transform our cities into net productive systems. Instead of focussing on reducing the impact of cities, we should use urban ecology and secondary production so that the city becomes a net positive resource generator. This is far more ambitious than existing projects that aim to create individual districts that are net zero (e.g. that produce as much energy as they consume, through local solar and wind generation). The next goal should be productive cities: cities that produce more resources than they consume; cities that process more waste than they produce.

Jeb then went on to crunch the numbers for a number of different types of resource (energy, food, metals, nitrogen), to demonstrate how a productive city might fill the gap between rising demand and declining supply:

Energy demand. Current European consumption is around 74GJ/capita. Imagine by 2050, we have 9 billion people on the planet, all living like Europeans do now – we’ll need 463EJ to supply them all. Plot this growth in demand over time, and you have a wedge analysis. Using IEA numbers of projected growth in renewable energy supply, to provide the wedges, there’s still a significant shortfall. We’ll need to close the gap via urbanrenewable energy generation, using community designs of the type piloted in Hannover. Cities have to become net producers of energy.

Here’s the analysis (click each chart for full size):

Food. We can do a similar wedge analysis for food. Current food production globally produces around 2,800kcal/captia. But as the population grows, this current level of production produces steadily less food per person. Projected increases in crop yields, crop intensity, and conversion of additional arable land, and reduction of waste would still leave a significant gap if we wish to provide a comfortable 3100kcal/capita. While urban agriculture is unlikely to displace rural farm production, it can play a crucial role in closing the gap between production and need, as the population grows. For example, Havana has a diversified urban agriculture that supplies close to 75% of vegetables from within the urban environment. Vancouver has been very strategic about building its urban agricultural production, with one out of every seven jobs in Vancouver in food production.

Other examples include landfill mining to produce iron and other metals, and urban production of nitrogen fertilizer from municipal biosolids.

In summary, we’ve always underestimated just how much we can transform cities. While we remain stuck in a mindset that cities are extraction engines, we will miss opportunities for more radical re-imagings of the role of global cities. So a key research challenge is to develop a new post-“ecological footprint” analysis. There are serious issues of scaling and performance measurement to solve, and at every scale there are technical, policy, and social challenges. But as cities house ever more of the growing population, we need this kind of bold thinking.

My first year climate course

12. October 2012 · 4 comments · Categories: climate science

My first year seminar course, PMU199 Climate Change: Software, Science and Society is up and running again this term. The course looks at the role of computational models in both the science and the societal decision-making around climate change. The students taking the course come from many different departments across arts and science, and we get to explore key concepts in a small group setting, while developing our communication skills.

As an initial exercise, this year’s cohort of students have written their first posts for the course blog (assignment: write a blog post on any aspect of climate change that interests you). Feel free to comment on their posts, but please keep it constructive – the students get a chance to revise their posts before we grade them (and if you’re curious, here’s the rubric).

Incidentally, for the course this year, I’ve adopted Andrew Dessler’s new book, Introduction to Modern Climate Change as the course text. The book was just published earlier this year, and I must say, it’s by far the best introductory book on climate science that I’ve seen. My students tell me they really like the book (despite the price), as it explains concepts simply and clearly, and they especially like the fact that it covers policy and society issues as well as the science. I really like the discussion in chapter 1 on who to believe, in which the author explains that readers ought to be skeptical of anyone writing on this topic (including himself), and then lays out some suggestions for how to decide who to believe. Oh, and I love the fact that there’s an entire chapter later in the book devoted to the idea of exponential growth.

What is Climate Informatics?

21. September 2012 · 1 comment · Categories: climate informatics

I’ve been using the term Climate Informatics informally for a few years to capture the kind of research I do, at the intersection of computer science and climate science. So I was delighted to be asked to give a talk at the second annual workshop on Climate Informatics at NCAR, in Boulder this week. The workshop has been fascinating – an interesting mix of folks doing various kinds of analysis on (often huge) climate datasets, mixing up techniques from Machine Learning and Data Mining with the more traditional statistical techniques used by field researchers, and the physics-based simulations used in climate modeling.

I was curious to see how this growing community defines itself – i.e. what does the term “climate informatics” really mean? Several of the speakers offered definitions, largely drawing on the idea of the Fourth Paradigm, a term coined by Jim Gray, who explained it as follows. Originally, science was purely empirical. In the last few centuries, theoretical science came along, using models and generalizations, and in the latter half of the twentieth century, computational simulations. Now, with the advent of big data, we can see a fourth scientific research paradigm emerging, sometimes called eScience, focussed on extracting new insights from vast collections of data. By this view, climate informatics could be defined as data-driven inquiry, and hence offers a complement to existing approaches to climate science.

However, there’s still some confusion, in part because the term is new, and crosses disciplinary boundaries. For example, some people expected that Climate Informatics would encompass the problems of managing and storing big data (e.g. the 3 petabytes generated by the CMIP5 project, or the exabytes of observational data that is now taxing the resources of climate data archivists). However, that’s not what this community does. So, I came up with my own attempt to define the term:

I like this definition for three reasons. First, by bringing Information Science into the mix, we can draw a distinction between climate informatics and other parts of computer science that are relevant to climate science (e.g. the work of building the technical infrastructure for exascale computing, designing massively parallel machines, data management, etc). Secondly, information science brings with it a concern for the broader societal and philosophical questions of the nature of information and why people need it, a concern that’s often missing from computer science. Oh, and I also like this definition because I also work at the intersection of the three fields, even though I don’t really do data-driven inquiry (although I did, many years ago, write an undergraduate thesis on machine learning). Hence, it creates a slightly broader definition than just associating the term with the ‘fourth paradigm’.

Having defined the field this way, it immediately suggests that climate informatics should also concern itself with the big picture of how we get get beyond mere information, and start to produce knowledge and (hopefully) wisdom:

This diagram is adapted from a classic paper by Russ Ackoff “From Data to Wisdom”, Journal of Applied Systems Analysis, Volume 16, 1989 p 3-9. Ackoff originally had Understanding as one of the circles, but subsequent authors have pointed out that it makes more sense as one of two dimensions you move along as you make sense of the data, the other being ‘context’ or ‘connectedness’.

The machine learning community offers a number of tools primarily directed at moving from Data towards Knowledge, by finding patterns in complex datasets. The output of a machine learner is a model, but it’s a very different kind of model from the computational models used in climate science: it’s a mathematical model that describes the discovered relationships in the data. In contrast, the physics-based computational models that climate scientists build are more geared towards moving in the opposite direction, from knowledge (in the form of physical theories of climactic processes) towards data, as a way of exploring how well current theory explains the data. Of course, you can also run a climate model to project the future (and, presumably, help society choose a wise path into the future), but only once you’ve demonstrated it really does explain the data we already have about the past. Clearly the two approaches are complementary, and ought to be used together to build a strong bridge between data and wisdom.

Note: You can see all the other invited talks (including mine), at the workshop archive. You can also explore the visuals I used (with no audio) at Prezi (hint: use full screen for best viewing).

Don’t feed the trolls

08. August 2012 · 6 comments · Categories: blogging, conferences

It’s AGU abstract submission day, and I’ve just submitted one to a fascinating track organised by John Cook, entitled “Social Media and Blogging as a Communication Tool for Scientists”. The session looks like it will be interesting, as there are submissions from several prominent climate bloggers. I decided to submit an abstract on moderation policies for climate blogs:

Don’t Feed the Trolls: An analysis of strategies for moderating discussions on climate blogs
A perennial problem in any online discussion is the tendency for discussions to get swamped with non-constructive (and sometimes abusive) comments. Many bloggers use some form of moderation policy to filter these out, to improve the signal to noise ratio in the discussion, and to encourage constructive participation. Unfortunately, moderation policies have disadvantages too: they are time-consuming to implement, introduce a delay in posting contributions, and can lead to accusations of censorship and anger from people whose comments are removed.

In climate blogging, the problem is particularly acute because of the politicization of the discourse. The nature of comments on climate blogs vary widely. For example, on a blog focussed on the physical science of climate, comments on posts might include personal abuse, accusations of misconduct and conspiracy, repetition of political talking points, dogged pursuit of obscure technical points (whether related or not to the original post), naive questions, concern trolling (negative reactions posing as naive questions), polemics, talk of impending doom and catastrophe, as well as some honest and constructive questions about the scientific topic being discussed. How does one decide which of these comments to allow? And if some comments are to be removed, what should be done with them?

In this presentation, I will survey a number of different moderation strategies used on climate blogs (along with a few notable examples from other kinds of blogs), and identify the advantages and disadvantages of each. The nature of the moderation strategy has an impact on the size and kind of audience a blog attracts. Hence, the choice of moderation strategy should depend on the overall goals for the blog, the nature of the intended audience, and the resources (particularly time) available to implement the strategy.

Nobody invented the internet!

05. June 2012 · 3 comments · Categories: philosophy

There’s been plenty of reaction across the net this week in response to a daft NYT article that begins “Men invented the internet”. At BoingBoing, Xeni is rightly outraged at the way the article is written, and in response comes up with plenty of examples of the contributions women have made to the development of computing. Throughout the resulting thread, many commentators chip in with more examples. Occasionally a (male) commentator shows up and tries to narrow down the definition of “invented the internet” to show that, yes, it was a man who invented some crucial piece of technology that makes it work. These comments are very revealing of a particular (male!) mindset towards technology and invention.

The central problem in the discussion seems to have been missed entirely. The real problem word in that opening sentence isn’t the word “men”, it’s the word “invented”. The internet is an incredibly complex socio-technical system. The notion that any one person (or any small group of people) invented it is ludicrous. Over a period of decades, the various technologies, protocols and conventions that make the internet work gradually evolved, through the efforts of a huge number of people (men and women), through a remarkable open design process. The people engaged in this endeavour had to invent new social processes for sharing and testing design ideas and getting feedback (for example, the RFC). It was these social processes, as much as any piece of technology, that made the internet possible.

But we should go further, because the concept of “invented” is even more problematic. If you study how any modern device came to be, the idea that there is a unique point in space and time that can be called its “invention” is really just a fiction. Henry Petroski does a great job of demonstrating this, through his histories of every day objects such as pencils, cutlery, and so on. The technologies we rely on today all passed though a long history of evolution in the same way. Each new form is a variant of ones that have gone before, created to respond to a perceived flaw in its predecessors. Some of these new variants are barely different from others, others represent larger modifications. Many of these modifications are worse than the original, some are better for specific purposes (and hence may start a new niche), and occasionally a more generally useful improvement is made.

The act of pointing to these occasional, larger modifications, and choosing to label them as “the birth of the modern X”, or the “first X”, or “the invention of X”, is a purely a social construct. We do it because we’re anchored in the present, seeing only the outcomes of these evolutionary processes, and we make the same mistake that creationists make, of being unable to conceive of the huge variety of intermediate forms that came before, and the massive process of trial and error that selected a particular form to survive and prosper. And, through continued operation of that bias, we’ve been conditioned to think in terms of unique moments of “invention” (often accompanied by a caricature of the lonely inventor working late at night in the lab).

And one of the biggest differences between men and women, in terms of social behaviour, is that men tend to boast about their successes and identify winners, while women tend to acknowledge group contributions and downplay their own efforts. So it’s hardly surprising that our history books are more full of male “inventors” than female inventors – the very idea of looking for a unique person to call the “inventor” is largely a male concept. Not only that, but it’s overwhelmingly a rich white guys’ way of looking at the world. The rich and powerful get to make decisions about who gets the credit for stuff. Not surprisingly, rich and powerful white men tend to pick other white men to designate as the “inventor”, and marginalize the contributions of others, no matter who else contributed to the idea during its gestation.

Update: Jan 9, 2014: Thanks to my student, Elizabeth, I now know the term for this: it’s the Matthew Effect. The wikipedia page has lots of examples.

What is Science Literacy?

29. May 2012 · 5 comments · Categories: education

A few people today have pointed me at the new paper by Dan Kahan & colleagues (1), which explores competing explanations for why lots of people don’t regard climate change as a serious problem. I’ve blogged about Dan’s work before – their earlier studies are well designed, and address important questions. If you’re familiar with their work, the new study isn’t surprising. They find that people’s level of concern over climate change doesn’t correlate with their level of science literacy, but does correlate with their cultural philosophies. In this experiment, there was no difference in science literacy between people who are concerned about climate change and those who are not. They use this to build a conclusion that giving people more facts is not likely to change their minds:

A communication strategy that focuses only on transmission of sound scientific information, our results suggest, is unlikely to do that.

Which is reasonable advice, because science communication must address how different people see the world, and how people filter information based on their existing worldview:

Effective strategies include use of culturally diverse communicators, whose affinity with different communities enhances their credibility, and information-framing techniques that invest policy solutions with resonances congenial to diverse groups

Naturally, some disreputable newsites spun the basic finding as a “Global warming skeptics as knowledgeable about science as climate change believers, study says“. Which is not what the study says at all, because it didn’t measure what people know about the science.

The problem is that there’s an unacknowledged construct validity problem in the study. At the beginning of the paper, the authors talk about “the science comprehension thesis (SCT)”:

As members of the public do not know what scientists know, or think the way scientists think, they predictably fail to take climate change as seriously as scientists believe they should.

…which they then claim their study disproves. But when you get into the actual study design, they quickly switch to talking about science literacy:

We measured respondents’ science literacy with National Science Foundation’s (NSF) Science and Engineering Indicators. Focused on physics and biology (for example, ‘Electrons are smaller than atoms [true/false]’; ‘Antibiotics kill viruses as well as bacteria [true/false]’), the NSF Indicators are widely used as an index of public comprehension of basic science.

But the problem is, science comprehension cannot be measured by asking a whole bunch of true/false questions about scientific “facts”! All that measures is the ability to do well in a pub trivia quizzes.

Unfortunately, this mistake is widespread, and leads to an education strategy that fills students’ heads with a whole bunch of disconnected science trivia, and no appreciation for what science is really all about. When high school students learn chemistry, for example, they have to follow recipes from a textbook, and get the “right” results. If their results don’t match the textbooks, they get poor marks. When they’re given science tests (like the NSF one used in this study), they’re given the message that there’s a right and wrong answer to each question, and you just gotta know it. But that’s about as far from real science as you can get! It’s when the experiment gives surprising results that the real scientific process kicks in. Science isn’t about getting the textbook answer, it’s about asking interesting questions, and finding sound methods for answering them. The myths about science are ground into kids from an early age by people who teach science as a bunch of facts (3).

At the core of the problem is a failure to make the distinction that Maienschein points out between science literacy and scientific literacy (2). The NSF instrument measures the former. But science comprehension is about the latter – it…

…emphasizes scientific ways of knowing and the process of thinking critically and creatively about the natural world.

So, to Kahan’s interpretation of the results, I would add another hypothesis: we should actually start teaching people about what scientists do, how they work, and what it means to collect and analyze large bodies of evidence. How results (yes, even published ones) often turn out to be wrong, and what matters is the accumulation of evidence over time, rather than any individual result. After all, with Google, you can now quickly find a published result to support just about any crazy claim. We need to teach people why that’s not science.

Update (May 30, 2012): Several people suggested I should also point out that the science literacy test they used is for basic science questions across physics and biology; they did not attempt to test in any way people’s knowledge of climate science. So that seriously dents their conclusion: The study says nothing about whether giving people more facts about climate science is likely to make a difference.

Update2 (May 31, 2012): Some folks on Twitter argued with my statement “concern over climate change doesn’t correlate with … level of science literacy”. Apparently none of them have a clue how to interpret statistical analysis in the behavioural sciences. Here’s Dan Kahan on the topic. (h/t to Tamsin).

References:

(1) Kahan, D., Peters, E., Wittlin, M., Slovic, P., Ouellette, L., Braman, D., & Mandel, G. (2012). The polarizing impact of science literacy and numeracy on perceived climate change risks Nature Climate Change DOI: 10.1038/nclimate1547
(2) Maienschein, J. (1998). Scientific Literacy Science, 281 (5379), 917-917 DOI: 10.1126/science.281.5379.917
(3) William F. McComas (1998). The Principle Elements of the Nature of Science: Dispelling the Myths The Nature of Science in Science Education

Systems thinking and Genetically Modified food

25. May 2012 · 10 comments · Categories: systems thinking

I’ve been following a heated discussion on twitter this past week about a planned protest on Sunday in the UK, in which environmentalists plan to destroy a crop of genetically modified wheat being grown as part of a scientific experiment at Rothamsted, in Hertfordshire (which is, incidentally, close to where I grew up). Many scientists I follow on twitter are incensed, calling the protest anti-science. And some worry that it’s part of a larger anti-science trend in which the science on issues such as climate change gets ignored too. In return, the protesters are adamant that the experiment should not be happening, no matter what potential benefits the research might bring.

I’m fascinated by the debate, because it seems to be a classic example of the principle of complementarity in action, with each group describing things in terms of different systems, and rejecting the others’ position because it makes no sense within their own worldview. So, it should make a great case study for applying boundary critique, in which we identify the system that each group is seeing, and then explore where they’ve chosen to draw the boundaries of that system, and why. I think this will make a great case study for my course next month.

I’ve identified eight different systems that people have talked about in the debate. This is still something of a work in progress (and I hope my students can extend the analysis). So here they are, and for each some initial comments towards a boundary critique:

A system of scientists doing research. Many scientists see the protests as nothing more than irrational destruction of research. The system that motivates this view is a system of scientific experimentation, in which expert researchers choose problems to work on, based on their expectation that the results will be interesting and useful in some way. In this case, the GM trials are applied research – there is an expectation that the modified wheat might lead to agricultural improvements (e.g. through improved yield, or reduced need for fertilizer or pesticide). Within this system, science is seen as a neutral pursuit of knowledge, and therefore, attempts to disrupt experiments must be “anti-knowledge”, or “anti-science”. People who operate within this system tend to frame the discussion in terms of an attack on a particular group of researchers (on twitter, they’ve been using the hashtag #dontdestroyresearch), and they ask, pointedly, whether green politicians and groups condone or condemn the destruction. (The irony here is that the latter question itself is, itself, unscientific – it’s a rhetorical device used in wedge politics – but few of the people using it acknowledge this). Questions about whether certain kinds of research are ethical, or who might yield the benefits from this research lie outside the boundary of this system, and so are not considered. It is assumed that the researchers themselves, as experts, have made those judgments well, and that the research itself is not, and cannot be, a political act.
A system of research ethics and risk management. If we expand the boundaries of system 1 a little, we see a system of processes by which scientific experiments are assessed for how they manage the risks they pose to be public. Scientific fields differ in their sophistication for how they arrange this system. In the physical sciences, the question often doesn’t arise, because the the research itself carries no risk. But in medical and social sciences, processes have arisen for making this judgement, sometimes in response to a disaster or a scandal. Most research institutions have set up Internal Review Boards (IRBs) who must approve (or prevent) research studies that poses a risk to people or ecosystems. My own research often strays into behavioural science, so I frequently have to go though our ethics approval process. The approvals process is usually frustrating, and I’m often surprised at some of the modifications the ethics board asks me to make, because my assessment of the risk is different to theirs. However, if I take a step back, I can see that both the process and the restrictions it places on me are necessary, and that I’m definitely not the right person to make judgements about the risks I might impose on others in my research. The central question is usually one of beneficence: does the value of the knowledge gained outweigh any potential risk to participants or others affected by the study? Some research clearly should not happen, because the argument for beneficence is too weak. In this view, the Rothamsted protest is really about democratic control of the risk assessment process. If all stakeholders aren’t included, and the potential impact on them is not taken seriously, they lose faith in the scientific enterprise itself. In the case of GMOs, there’s a widespread public perception (in the UK) that the interests of large corporations who stand to profit from this research are being allowed to drive the approvals process, and that the researchers themselves are unable to see this because they’re stuck in system 1. I’ve no idea how true this is for GMO research, but there’s plenty of evidence that’s it’s become a huge problem in pharmaceutical research. Medical research organizations have, in the last few years, taken significant steps to reduce the problem, e.g by creating registers of trials to ensure negative results don’t get hidden. The biotech research community appear to be way behind on this, and much research still gets done behind the veil of corporate secrecy. (The irony here is that the Rothamsted trials are publicly funded, and results will be publicly available, making it perhaps the least troublesome biotech research with respect to corporate control. However, that visibility makes it an easy target, and hence, within this system, the protest is really an objection to how the government ran the risk assessment and approval process for this experiment).
A system of ecosystems and contaminants that weaken them. Some of the protesters are focused more specifically on the threat that this and similar experiments might pose on neighbouring ecosystems. In this view, GMOs are a potential contaminant, which, once released into the wild cannot ever be recalled. Ecosystems are complex systems, and we still don’t understand all the interactions that take place within them, and how changing conditions can damage them. Previous experimentation (e.g. the introduction of non-native species, culls of species regarded as pests, etc), have often been disastrous, because of unanticipated system interactions. Within this system, scientists releasing GMOs into the wild are potentially repeating these mistakes of the past, but on a grander scale, because a GMO represents a bigger step change within the system than, say, selective breeding. Because these ecosystems have non-linear dynamics, bigger step changes aren’t just a little more risky than small step changes; they risk hitting a threshold and causing ecosystem collapse. People who see this system tend to frame the discussion in terms of the likelihood of cross-contamination by the GMO, and hence worry that no set of safeguards by the researchers is sufficient to guarantee the GMO won’t escape. Hence, they object to the field trials on principle. This trial is therefore, potentially, the thin end of the wedge, a step towards lifting the wider ban on such trials. If this trial is allowed to go ahead, then others will surely follow, and sooner or later, various GMOs will escape with largely unpredictable consequences for ecosystems. As the GMOs are supposed to have a competitive advantage of other related species, once they’ve escaped, they’re likely to spread, in the same way that invasive species did. So, although the researchers in this experiment may have taken extensive precautions to prevent cross-contamination, such measures will never be sufficient to guarantee protection, and indeed, there’s already a systematic pattern of researchers underestimating the potential spread of GMO seeds (e.g. through birds and insects), and of course, they routinely underestimate the likelihood of human error. Part of the problem here is that the researchers themselves are biased in at least two ways: they designed the protection measures themselves, so they tend to overestimate their effectiveness, and they believe their GMOs are likely to be beneficial (otherwise they wouldn’t be working on them), so they downplay the risk to ecosystems if they do escape. Within this system, halting this trial is equivalent to protecting the ecosystems from risky contamination. (The irony here is that a bunch of protesters marching into the field to destroy the crop is likely to spread the contamination anyway. The protesters might rationalize it by saying this particular trial is more symbolic, because the risk from any one trial is rather low; instead the aim is to make it impossible for future trials to go ahead)
A system of intellectual property rights and the corresponding privatization of public goods. Some see GMO research as part of a growing system of intellectual property rights, in which large corporations gain control of who can grow which seeds and when. In Canada, this issue became salient when Monsanto tried suing farmers who were found to have their genetically modified corn planted in their fields, despite the fact that those farmers had never planted them (it turned out the seeds were the result of cross-contamination from other fields, something that Monsanto officially denies is possible). By requiring farmers to pay a licence fee each year to re-plant their proprietary seeds, these companies create a financial dependency that didn’t exist when farmers were able to save seeds to be replanted. Across developing countries, there is growing concern that agribusiness is gaining too much control of local agriculture, creating a market in which only their proprietary seeds can be planted, and hence causing a net outflow of wealth from countries that can least afford it to large multi-national corporations. I don’t see this view playing a major role in the UK protests this week, although it does come up in the literature from the protest groups, and is implicit in the name of the protest group: Take The Flour Back.
An economic system in which investment in R&D is expected to boost the economy. This is the basic capitalist system. Companies that have the capital invest in research into new technologies (GMOs) that can potentially bring big returns on investment for biotech corporations. This is almost certainly the UK government’s perspective on the trials at Rothamsted – the research should be good for the economy. It’s also perhaps the system that motivates some of the protesters, especially where they see this system exacerbating current inequalities (big corporations get richer, everyone else pays more for their food). Certainly, economic analysis of the winners and losers from GM technology demonstrate that large corporations gain, and small-scale farmers lose out.
A system of global food supply and demand, in which a growing global population, and a fundamental limit on the land available for agriculture, place serious challenges on how to achieve a better match of food consumption to food production. In the past, we solved this problem through two means: expanding the amount of land under cultivation, and through the green revolution, in which agricultural yields were increased by industrialization of the agricultural system and the wide-scale use of artificial fertilizers. GMOs are (depending on who you ask) either the magic bullet that will allow us to feed 9 billion people by mid-century, or, more modestly, one of many possible solutions that we should investigate. In this system, the research at Rothamsted is seen as a valuable step towards solving world hunger, and so protesting against it is irrational. The irony here is that improving agricultural yields is probably the least important part of the challenge of feeding 9 billion people: there is much more leverage to be had in solving problems of food distribution, reducing wastage, and reducing the amount of agricultural land devoted to non-foods.
A system of potential threats to human health and well-being. Some see GMOs as a health issue. Potential human health effects include allergies, and cross-species genetic transfer, although scientists dismiss both, citing a lack of evidence. While there is some (disputed) evidence of such health risks already occurring, on balance this is more a concern about unpredictable future impacts, rather than what has already happened, which means an insistence on providing evidence is irrelevant: a bad outcome doesn’t have to have already occurred for us to take the risk seriously. If we rely on ever more GMOs to drive the global agricultural system, sooner or later we will encounter such health problems, most likely through increased allergic reaction. Allergies themselves have interesting systemic properties – they arise when the body’s normal immune system, doing it’s normal thing, ends up over-reacting to a stimulus (e.g. new proteins) that is otherwise harmless. The concern here, then, is that the reinforcing feedback loop of ever more GM plant variants means that, sooner or later, we will cross a threshold where there is an impact on human health. People who worry about this system tend to frame the discussion using terms such as “Frankenfoods“, a term that is widely derided by biotech scientists. The irony here is that by dismissing such risks entirely, the scientists reduce their credibility in the eyes of the general public, and end up seeming even more like Dr Frankenstein, oblivious to their own folly.
A system of sustainable agriculture, with long time horizons. In this system, short term improvements in agricultural yield are largely irrelevant, unless the improvement can be demonstrated to be sustainable indefinitely without further substantial inputs to the system. In general, most technological fixes fail this test. The green revolution was brought about by a massive reliance on artificial fertilizer, derived from fossil fuels. As we hit peak oil, this approach cannot be sustained. Additionally, the approach has brought its own problems, including a massive nitrogen pollution of lakes and coastal waters, and poorer quality soils, and of course, the resulting climate change from the heavy use of fossil fuels. In this sense, technological fixes provide short term gains in exchange for a long term debt that must be paid by future generations. In this view, GMOs are seen as an even bigger step in the wrong direction, as they replace an existing diversity in seed-stocks and farming methods with industrialized mono-cultures, and divert attention away from the need for soil conservation, and long-term sustainable farming practices. In this system, small scale organic farming is seen as the best way of improving the resilience of the global food production. While organic farming sometime (but not always!) means lower yields, it reduces dependency on external inputs (e.g. artificial fertilizers and pesticides), and increases diversity. Systems with more diverse structures tend to be more resilient in the face of new threats, and the changing climates over the next few decades will severely test the resilience of our farming methods in many regions of the world. The people who worry about this system point to failures of GMOs to maintain their resistance to pests. Here, you get a reinforcing feedback loop in which you need ever more advances in GMO technology to keep pace with the growth of resistance within the ecosystem, and with each such advance, you make it harder for non-GMO food varieties to survive. So while most proponents of GMOs see them as technological saviours, in the long term it’s likely they actually reduce the ability of the global agricultural system to survive the shocks of climate change.

Systems theory leads us to expect that these systems will interact in interesting ways, and indeed they do. For example, systems 6 and 8 can easily be confused as having the same goal, but in fact, because the systems have very different temporal scales, they can end up being in conflict: short-term improvements to agricultural yield can lead to long term reduction of sustainability and resilience. Systems 6 and 7 can also interfere – it’s been argued that the green revolution reduced world starvation and replaced it with widespread malnutrition, as industrialization of food production gives us fewer healthy food choices. Systems 1 and 4 are often in conflict, and are leading to ever more heated debates over open access to research results. And of course, one of the biggest worries of some of the protest groups is the interaction between systems 2 and 5: the existence of a large profit motive tends to weaken good risk management practices in biotech research.

Perhaps the most telling interaction is the opportunity cost. While governments and corporations, focusing on systems 5 & 6, pour funding and effort into research into GMOs, other, better solutions to long term sustainability and resilience, required in system 8, become under-invested. More simply: if we’re asking the wrong question about the benefit of GMOs, we’ll make poor decisions about whether to pursue them. We should be asking different questions about how to feed the world, and resources put into publicly funded GMO research tend to push us even further in the wrong direction.

So where does that leave the proposed protests? Should the trials at Rothamsted be allowed to continue, or do the protesters have the right to force an end to the experiment, by wilful destruction if necessary? My personal take is that the experiment should be halted immediately, preferably by Rothamsted itself, on the basis that it hasn’t yet passed the test for beneficence in a number of systems. The knowledge gain from this one trial is too small to justify creating this level of societal conflict. I’m sure some of my colleague will label me anti-science for this position, but in fact, I would argue that my position here is strongly pro-science: an act of humility by scientists is far more likely to improve the level of trust that the public has in the scientific community. Proceeding with the trial puts public trust in scientists further at risk.

Let’s return to that question of whether there’s an analogy between people attacking the biotech scientists and people attacking climate scientists. If you operate purely within system 1, the analogy seems compelling. However, it breaks down as soon as you move to system 2, because the risks have opposite signs. In the case of GMO food trials, the research itself creates a risk; choosing not to do the research at all (or destroying it if someone else tries it) is an attempt to reduce risk. In the case of climate science, the biggest risks are on the business-as-usual scenario. Choosing to do the research itself poses no additional risk, and indeed reduces it, because we come to understand more about how the climate system works.

The closest analogy in climate science I can think of is the debate over geo-engineering. Many climate scientists objected to any research being done on geo-engineering for many years, for exactly the reason many people object to GMO research – because it diverts attention away from more important things we should be doing, such as reducing greenhouse gas emissions. A few years back, the climate science community seems to have shifted perspective, towards the view that geo-engineering is a desperate measure that might buy us more time to get emissions under control, and hence research is necessary to find out how well it works. A few geo-engineering field trials have already happened. As these start to gain more public attention, I would expect the protests to start in earnest, along with threats to destroy the research. And it will be for all the same reasons that people want to destroy the GM wheat trials at Rothamsted. And, unless we all become better systems thinkers, we’ll have all the same misunderstandings.

Update (May 29, 2012): I ought to collect links to thought provoking articles on this. Here are some:

Imagine Toronto as a Better City

24. May 2012 · Write a comment · Categories: sustainability

There’s a fascinating piece up this week on The Grid on how to make Toronto a better city. They asked a whole bunch of prominent people for ideas, each to be no longer than 200 words. The ideas didn’t necessarily have to be practical, but would be things to make us think. Some of them are wacky, some are brilliant, and some are both. My favourites are:

Give people alternative ways to pay their dues, e.g. instead of taxes, struggling artists donate public art, etc. (Seema Jethalal);
Hold a blackout holiday twice a year, to mimic the sense of connectness we all got when the power grid went down in 2003 (Carlyle Jansen)
Use ranked ballots for all municipal elections (Dave Meslin)
Banish all outdoor commercial ads (Sean Martindale)
Ban parking on all main streets (Chris Hume)
Build a free wireless internet via decentralized network sharing (Jesse Hirsh)
Make the TTC (our public transit) free (David Mirvish)

Better yet, they asked for more suggestions from readers. Here are mine:

Safe bike routes to schools. Every school should be connected to a network of safe bike paths for kids. Unlike the city’s current bike network, these bike baths should avoid main roads as much as possible: bike lanes on main roads are not safe for kids. Instead they should go via residential streets, parks, and marginal spaces, and physically separate the bikes from all vehicle traffic. These routes should provide uninterrupted links from sheltered bike parking at each school all the way through the residential neighbourhoods that each school serves. They should also provide a larger network, linking each school with neighbouring schools, for families where the kids go to different local schools, and where kids use services (e.g. pools) in other local schools.

Advantages: kids get exercise biking to school, gain some independence from parents, and become better connected with their environment. Traffic congestion and pollution at school drop-off and pickup times would drop. To build such a network, we would have to sacrifice some on-street parking in residential streets. However, a complete network of such bike paths could become a safer alternative to the current bike lanes on main streets, thus freeing up space on main streets.

and:

Car-free blocks on streetcar routes. On each streetcar route through the city, select individual blocks (i.e. stretches between adjacent cross-streets) at several points along each route, and close these stretches to all other motorized vehicle traffic. Such blocks would only allow pedestrians, bikes and streetcars. The sidewalks would then be extended for use as patios by cafes and restaurants. Delivery vehicles would still be permitted, perhaps only at certain times of day.

The aim is to discourage other traffic from using the streets that our streetcars run on as major commuting corridors through the city, and to speed up the flow of streetcars. The blocks selected to pedestrianize would be those where there is already a lively street life, with existing cafes, etc. Such blocks would become desirable destinations for shoppers, diners and tourists.

Sustainability Visioning Symposium

18. May 2012 · Write a comment · Categories: sustainability

I’ve been working for the past couple of months with the Cities Centre and the U of T Sustainability Office to put together a symposium on sustainability, where we pose the question “What role should the University of Toronto play in the broader challenge of building a sustainable city?”. We now finally have all the details in place:

An Evening Lecture, on the evening of June 13, 6pm to 9pm, at Innis Town Hall, featuring Bob Willard, author of “The Sustainability Advantage”, Tanzeel Merchant, of the Ontario Growth Secretariat and Heritage Toronto, and John Robinson, Director of the UBC Sustainability initiative and the Royal Canadian Geographical Society’s Canadian Environmental Scientist of the Year.
A full day visioning workshop on June 14, 9am to 5pm, at the Debates Room, Hart House. With a mix of speakers and working group sessions, the goal will be to map out a vision for sustainability at U of T, that brings together research, teaching and operations at the University, and explores how we can use the University as a “living lab” to investigate challenges in urban sustainability.

And it’s free. Register here!

Science via Twitter

17. May 2012 · 3 comments · Categories: collaborative science

On my trip to Queens University last week, I participated in a panel session on the role of social media in research. I pointed out that tools like twitter provide a natural extension to the kinds of conversations we usually only get to have at conferences – the casual interactions with other researchers that sometimes lead to new research questions and collaborations.

So, with a little help from Storify, here’s an example…

In which we see and example of how Twitter can enable interesting science, and understand a little about the role of existing social networks in getting science done.

http://storify.com/SMEasterbrook/science-via-twitter

Rotating the Question

06. May 2012 · 3 comments · Categories: climate science

At the CMIP5 workshop earlier this week, one of Ed Hawkins‘ charts caught my eye, because he changed how we look at model runs. We’re used to seeing climate models used to explore the range of likely global temperature responses under different future emissions scenarios, and the results presented as a graph of changing temperature over time. For example, this iconic figure from the last IPCC assessment report (click for the original figure and caption at the IPCC site):

These graphs tend to focus too much on the mean temperature response in each scenario (where ‘mean’ means ‘the multi-model mean’). I tend to think the variance is more interesting – both within each scenario (showing differences in the various CMIP3 models on the same scenarios), and across the different scenarios (showing how our future is likely to be affected by the energy choices implicit in each scenario). A few months ago, I blogged about the analysis that Hawkins and Sutton did on these variabilities, to explore how the different sources of uncertainty change as you move from near term to long term. The analysis shows that in the first few decades, the differences in the models dominate (which doesn’t bode well for decadal forecasting – the models are all over the place). But by the end of the century, the differences between the emissions scenarios dominates (i.e. the spread of projections from the different scenarios is significantly bigger than the disagreements between models). Ed presented an update on this analysis for the CMIP5 models this week, which looks very similar.

But here’s the new thing that caught my eye: Ed included a graph of temperature responses tipped on its side, to answer a different question: how soon will the global temperature exceed the policymaker’s adopted “dangerous” threshold of 2°C, under each emissions scenario. And, again, how big is the uncertainty? This idea was used in a paper last year by Joshi et. al., entitled Projections of when temperature change will exceed 2 °C above pre-industrial levels. Here’s their figure 1:

Figure 1 from Joshi et al, 2011

By putting the dates on the Y-axis and temperatures on the X-axis, and cutting off the graph at 2°C, we get a whole new perspective on what the models runs are telling us. For example, it’s now easy to see that in all these scenarios, we pass the 2°C threshold well before the end of the century (whereas the IPCC graph above completely obscures this point), and under the higher emissions scenarios, we get to 3°C by the end of the century.

A wonderful example of how much difference the choice of presentation makes. I guess I should mention, however, that the idea of a 2°C threshold is completely arbitrary. I’ve asked many different scientists where the idea came from, and they all suggest it’s something the policymakers dreamt up, rather than anything arising out of scientific analysis. The full story is available in Randalls, 2011, “History of the 2°C climate target”.

How Climate Science is Done

02. May 2012 · 1 comment · Categories: climate modeling

In the talk I gave this week at the workshop on the CMIP5 experiments, I argued that we should do a better job of explaining how climate science works, especially the day-to-day business of working with models and data. I think we have a widespread problem that people outside of climate science have the wrong mental models about what a climate scientist does. As with any science, the day-to-day work might appear to be chaotic, with scientists dealing with the daily frustrations of working with large, messy datasets, having instruments and models not work the way they’re supposed to, and of course, the occasional mistake that you only discover after months of work. This doesn’t map onto the mental model that many non-scientists have of “how science should be done”, because the view presented in school, and in the media, is that science is about nicely packaged facts. In reality, it’s a messy process of frustrations, dead-end paths, and incremental progress exploring the available evidence.

Some climate scientists I’ve chatted to are nervous about exposing more of this messy day-to-day work. They already feel under constant attack, and they feel that allowing the public to peer under the lid (or if you prefer, to see inside the sausage factory) will only diminish people’s respect for the science. I take the opposite view – the more we present the science as a set of nicely polished results, the more potential there is for the credibility of the science to be undermined when people do manage to peek under the lid (e.g. by publishing internal emails). I think it’s vitally important that we work to clear away some of the incorrect mental models people have of how science is (or should be) done, and give people a better appreciation for how our confidence in scientific results slowly emerges from a slow, messy, collaborative process.

Giving people a better appreciation of how science is done would also help to overcome some of games of ping pong you get in the media, where each new result in a published paper is presented as a startling new discovery, overturning previous research, and (if you’re in the business of selling newspapers, preferably) overturning an entire field. In fact, it’s normal for new published results to turn out to be wrong, and most of the interesting work in science is in reconciling apparently contradictory findings.

The problem is that these incorrect mental models of how science is done are often well entrenched, and the best that we can do is to try to chip away at them, by explaining at every opportunity what scientists actually do. For example, here’s a mental model I’ve encountered from time to time about how climate scientists build models to address the kinds of questions policymakers ask about the need for different kinds of climate policy:

This view suggests that scientists respond to a specific policy question by designing and building software models (preferably testing that the model satisfies its specification), and then running the model to answer the question. This is not the only (or even the most common?) layperson’s view of climate modelling, but the point is that there are many incorrect mental models of how climate models are developed and used, and one of the things we should strive to do is to work towards dislodging some of these by doing a better job of explaining the process.

With respect to climate model development, I’ve written before about how models slowly advance based on a process that roughly mimics the traditional view of “the scientific method” (I should acknowledge, for all the philosophy of science buffs, that there really isn’t a single, “correct” scientific method, but let’s keep that discussion for another day). So here’s how I characterize the day to day work of developing a model:

Most of the effort is spent identifying and diagnosing where the weaknesses in the current model are, and looking for ways to improve them. Each possible improvement then becomes an experiment, in which the experimental hypothesis might look like:

“if I change <piece of code> in <routine>, I expect it to have <specific impact on model error> in <output variable> by <expected margin> because of <tentative theory about climactic processes and how they’re represented in the model>”

The previous version of the model acts as a control, and the modified model is the experimental condition.

But of course, this process isn’t just a random walk – it’s guided at the next level up by a number of influences, because the broader climate science community (and to some extent the meteorological community) are doing all sorts of related research, which then influences model development. In the paper we wrote about the software development processes at the UK Met Office, we portrayed it like this:

But I could go even broader and place this within a context in which a number of longer term observational campaigns (“process studies”) are collecting new types of observational data to investigate climate processes that are still poorly understood. This then involves the interaction several distinct communities. Christian Jakob portrays it like this:

Although the point of Jakob’s paper is to argue that the modelling and process studies communities don’t currently do enough of this kind of interactions, so there’s room for improvement in how the modelling influences the kinds of process studies needed, and how the results from process studies feed back into model development.

So, how else should we be explaining the day-to-day work of climate scientists?

Some CMIP5 statistics

29. April 2012 · 3 comments · Categories: climate modeling

I’m attending a workshop this week in which some of the initial results from the Fifth Coupled Model Intercomparison Project (CMIP5) will be presented. CMIP5 will form a key part of the next IPCC assessment report – it’s a coordinated set of experiments on the global climate models built by labs around the world. The experiments include hindcasts to compare model skill on pre-industrial and 20th Century climate, projections into the future for 100 and 300 years, shorter term decadal projections, paleoclimate studies, plus lots of other experiments that probe specific processes in the models. (For more explanation, see the post I wrote on the design of the experiments for CMIP5 back in September).

I’ve been looking at some of the data for the past CMIP exercises. CMIP1 originally consisted of one experiment – a control run with fixed forcings. The idea was to compare how each of the models simulates a stable climate. CMIP2 included two experiments, a control run like CMIP1, and a climate change scenario in which CO2 levels were increased by 1% per year. CMIP3 then built on these projects with a much broader set of experiments, and formed a key input to the IPCC Fourth Assessment Report.

There was no CMIP4, as the numbers were resynchronised to match the IPCC report numbers (also there was a thing called the Coupled Carbon Cycle Climate Model Intercomparison Project, which was nicknamed C4MIP, so it’s probably just as well!), so CMIP5 will feed into the fifth assessment report.

So here’s what I have found so far on the vital statistics of each project. Feel free to correct my numbers and help me to fill in the gaps!

	CMIP (1996 onwards)	CMIP2 (1997 onwards)	CMIP3 (2005-2006)	CMIP5 (2010-2011)
Number of Experiments	1	2	12	110
Centres Participating	16	18	15	24
# of Distinct Models	19	24	21	45
# of Runs (Models X Expts)	19	48	211	841
Total Dataset Size	1 Gigabyte	500 Gigabyte	36 TeraByte	3.3 PetaByte
Total Downloads from archive	??	??	1.2 PetaByte
Number of Papers Published	47		595
Users	??	??	6700

[Update:] I’ve added a row for number of runs, i.e. the sum of the number of experiments run on each model (in CMIP3 and CMIP5, centres were able to pick a subset of the experiments to run, so you can’t just multiply models and experiments to get the number of runs). Also, I ought to calculate the total number of simulated years that represents (If a centre did all the CMIP5 experiments, I figure it would result in at least 12,000 simulated years).

Oh, one more datapoint from this week. We came up with an estimate that by 2020, each individual experiment will generate an Exabyte of data. I’ll explain how we got this number once we’ve given the calculations a bit more of a thorough checking over.

Climate modeling in an open, transparent world

20. April 2012 · 5 comments · Categories: AGU fall meeting 2012, climate informatics

As today is the deadline for proposing sessions for the AGU fall meeting in December, we’ve submitted a proposal for a session to explore open climate modeling and software quality. If we get the go ahead for the session, we’ll be soliciting abstracts over the summer. I’m hoping we’ll get a lively session going with lots of different perspectives.

I especially want to cover the difficulties of openness as well as the benefits, as we often hear a lot of idealistic talk on how open science would make everything so much better. While I think we should always strive to be more open, it’s not a panacea. There’s evidence that open source software isn’t necessarily better quality, and of course, there’re plenty of people using lack of openness as a political weapon, without acknowledging just how many hard technical problems there are to solve along the way, not least because there’s a lack of consensus over the meaning of openness among it’s advocates.

Anyway, here’s our session proposal:

TITLE: Climate modeling in an open, transparent world

AUTHORS (FIRST NAME INITIAL LAST NAME): D. A. Randall¹, S. M. Easterbrook⁴, V. Balaji², M. Vertenstein³

INSTITUTIONS (ALL): 1. Atmospheric Science, Colorado State University, Fort Collins, CO, United States. 2. Geophysical Fluid Dynamics Laboratory, Princeton, NJ, United States. 3. National Center for Atmospheric Research, Boulder, CO, United States. 4. Computer Science, University of Toronto, Toronto, ON, Canada.

Description: This session deals with climate-model software quality and transparent publication of model descriptions, software, and results. The models are based on physical theories but implemented as software systems that must be kept bug-free, readable, and efficient as they evolve with climate science. How do open source and community-based development affect software quality? What are the roles of publication and peer review of the scientific and computational designs in journals or other curated online venues? Should codes and datasets be linked to journal articles? What changes in journal submission standards and infrastructure are needed to support this? We invite submissions including experience reports, case studies, and visions of the future.