Random Hacks of Kindness

30. October 2009 · Write a comment · Categories: advocacy, software tools

Random Hacks of Kindness is a codejam sponsored by the World Bank, Google, Microsoft and Yahoo!, aimed at building useful software for important social/humanitarian causes. The upcoming event in the Bay Area in November is focussed on software for disaster relief.

However, they’re also proposing to run a 4-day codejam at the COP15 meeting in Copenhagen in December, aimed at building useful software for tackling climate change. I’ve submitted a few ideas of my own, plus our categorization of software challenges. Here’s some of my suggestions:

  • Make the IPCC website more accessible. E.g. provide a visual index of the figures and charts in the reports; develop “guided tours” through the material for different kinds of users, based on their various interests; provide pointers into key sections that respond to common misunderstandings
  • Provide simple dynamic visualizations of the key physical processes. Along the lines of the tutorial developed by John Sterman, but perhaps with less text and more freedom to play with the model.
  • Provide a simpler, web-based interface to the Java Climate model, that allows policymakers to quickly see the effects of different policy options.

What else?

Debunking myths or teaching the basics?

27. October 2009 · 2 comments · Categories: advocacy, climate science

Each time you encounter someone trying to claim human-induced global warming is a myth (e.g. because “Mars is warming too!”), you can save a lot of time and energy by just saying, oh yes, that’s myth #16 on the standard list of misunderstandings about climate change. Here’s the list, lovingly and painstakingly put together by John Cook.

Once you’ve got that out of the way, you can then challenge your assailant to identify a safe level of carbon dioxide in the atmosphere, and to get them to give evidence to justify that choice. If they don’t feel qualified to answer this question, then you get to a teachable moment. Take the opportunity to teach your assailant the difference between greenhouse gas emissions and greenhouse gas concentrations. That’s the single most important thing they have to understand. Here’s why:

  • We know that the earth warms by somewhere between 2 and 4.5°C (with a best estimate of about 3°C) for each doubling of CO2 concentrations in the atmosphere (this was first calculated over 100 years ago. The number has been refined a little as we’ve come to understand the physical processes better, but only within a degree or two)
  • CO2 is unlike any other pollutant: once it’s in the atmosphere it stays there for centuries (more specifically, it stays in the carbon cycle, being passed around between plants, soil, oceans, and atmosphere. But anyway, it only ever goes away when it eventually gets laid down as a new fossil layer, e.g. at the bottom of the ocean).
  • The earth’s temperature only responds slowly to changes in the level of greenhouse gases in the atmosphere. That means that even though we’ve seen warming of around 0.7°C over the last century, we’re still owed at least that much again due to the CO2 we have already added to the atmosphere.
  • The temperature is not determined by the amount of CO2 we emit; it’s determined by the total accumulation in the atmosphere – i.e. how thick the “blanket” is.
  • Because the carbon stays there for centuries, all new emissions increase the concentration, thus compounding the problem. The only sustainable level of net greenhouse gas emissions from human activities is zero.
  • If we ever manage to get to the point where net emissions of greenhouse gases from human activities is zero, the planet will eventually (probably, over centuries) return to pre-industrial atmospheric concentration levels (about 270 parts per million), as the carbon gets reburied. During this time, the earth will continue to warm.
  • Net emissions is, of course, the difference between gross emissions and any carbon we manage to remove from the system artificially. As no technology currently exists today for reliably and permanently removing carbon from the system, it would be prudent to aim for zero gross emissions. And the quicker we do it, the less the planet will warm in the meantime.
  • And 3°C global average temperature is about the difference between the last ice age (which ended about 12,000 years ago) and today’s climate. In the last ice age there were ice sheets 0.5km thick over much of North America and Europe. Now imagine how different the earth will be with a further 3°C of warming.

Okay, so that might be a little bit too much for just one teachable moment. What we really need is a simple elegant tool to illustrate all this. Anyone up to building an interactive visualization? John Sterman tried, but I don’t rate his tool high on the usability scale.

Liveblogging Oopsla

27. October 2009 · Write a comment · Categories: conferences

This week I’m at OOPSLA, mainly for the workshop on software research and climate change, which went exceedingly well (despite some technical hiccups), and which I will blog once I get my notes together. Now I can relax and enjoy the rest of the conference.

Today, Tom Malone from MIT is giving a keynote talk to kick off the Onward! track. Tom and I chatted over dinner last night about his Climate Collaboratorium project, which is an attempt to meet many of the goals I’ve been discussing about creating tools to foster a constructive public discourse about climate change and its solutions. So I’m keen to hear what he has to say in his keynote.

13:32: Bernd Bruegge is giving an overview of what the Onward! conference (part of Oopsla) is about. This year, Onward! has grown from a track within Oopsla to being a fully fledged co-located conference of its own.

13:36: He’s now introducing Tom Malone. Which reminds me I ought to get his book, The Future of Work. Okay, now Tom’s up, and his talk is entitled “The Future of Collective Intelligence”. His opening question is “who here is happy?” – he got us to raise hands. Looks like the overwhelming majority of the audience are happy. His definition of collective intelligence deliberately dodges the question of what intelligence is: “Groups of individuals doing things collectively that seem intelligent”. Oh, and collective stupidity also happens, and one of the interesting research questions is to figure out why. By this definition, collective intelligence has existed for centuries, but recently new forms have arrived. For example, the way google searches work; and of course, wikipedia. For wikipedia, the key enabler was the organisational design, rather than the technology. More examples: digg, youtube, linux, prediction markets,…

His core research question is: “how can people and computers be connected so that collectively they act more intelligently than any person, group or computer has ever done before?” It’s an aspirational question, but to answer it we need a systematic attempt to understand collective intelligence (rather than just marveling at the various instances). First attempt was to identify and understand different species of collective intelligence. Realised that a more productive metaphor was to look for individual genes that are common across several difference species. Or put another way, what are the design patterns?

Four questions for underlying activity involved in every design pattern: who is doing it? what are they doing? why are they doing it? and how? Tom challenged us to think about what percentage of the intelligence, energy etc of the people in the organisation you are in right now (e.g. the Oopsla conference) are actually available to the organisation. Most people in the audience had low numbers: 30% or less, lots said less than 10%. Then he showed us a video of an experiment in which many people in a large room were collectively driving a simulated airplane. Everyone had a two sided reflective wand (red on one side, green on the other). Half the people controlled up and down, the other half controlled left and right. The video was hilarous, but also surprising in how well the audience did.

So, in this example, the “who” is the crowd. The crowd gene is useful when the locations of knowledge needed for a task are distributed over a crowd, and you’re not sure a priori where it is, but only works when attempts to subvert the task can be controlled in some way.

The why boils down to love, glory, or money. Appealing to love and glory, rather than money, can reduce costs (but not always). E.g. make the task fun and people will chose to do it anyway. More interestingly you can influence the direction of the task by offering money or glory for certain actions. But most people get the motivational factors wrong (or just don’t think about it).

The what often boils down to Create or Decide. Which then gives us four situations depending on whether the crowd does pieces of the task independently or not:

  1. Collection: ‘create’ task where the pieces are independent. Examples include Wikipedia, Sourceforge and Youtube. A special subcategory of the collection pattern is the competition pattern, where you only need a few of the pieces. Eg: TopCoder. and the Netflix prize. In the latter, the offer of a $1 million prize motivated many people to work on this for two years. Eventually, several competing teams combined their solutions, and collectively they met the goal of 10% improvement in Netflix’s movie recommender. Another example: the Matlab programming contests. In this one, the competing algorithms are made available to all teams, so they can take each other’s ideas and incorporate them. This mix of competition and collaboration appears to be strangely addictive to many people. Competition pattern useful when only one (or a few) good solutions are needed, and the motivation is clear.
  2. Collaboration: ‘create’ tasks where the pieces are dependent on one another. Wikipedia is also an example of this, because different edits to the same article are highly inter-dependent. These dependencies are coordinated in wikipedia through the talk pages. In Linux the coordination is through the discussion forums. Tom’s Climate Collaboratorium is another example. In this project, plans are proposed and discussed and voted on, and by combining the plans, the aim is to create better plans than would be available without the collaboration. Managing the inter-dependencies turns out to be the hard part of Collaboration projects. Most existing examples rely on manual coordination mechanisms. Interesting question is what automated support can be provided. Suggestions here include better explicit representations of the interdependencies. The Collaborative pattern works when a large scale task needs doing, there is no satisfatory way of dividing up the task into independent pieces, but there is a way to manage interdependencies.
  3. Group Decision: ‘decide’ tasks where the pieces are inter-dependent. Simple mechanisms include:
    • voting. Interesting example is a baseball team where the fans can do internet voting to decide batting order, pitching rotation, starting line-up, etc. They did this for one season and lost most of their games, possibly because fans of other teams sabotaged the voting. Similar attempt by a UK soccer team, but where you have to be an “owner” of the team (35 pounds per year) to vote. This team seems to be doing well. Another example: Kasparov vs. the world. Expected that Kasparov would win easily, in fact he later said it was the hardest game he ever played. One key was that the crowd could discuss their moves over a 24 hour period before voting on them.
    • consensus. This is what is used in wikipedia when there are disagreements.
    • averaging. Useful in some group estimation tasks. The averages of a large number of individual estimates are often more accurate than individual estimates. Another example: NASA clickworkers, used to get crowds of people to identify craters on photos of astronomical bodies. The averaging of many novices did pretty much the same as experts (and much more cheaply). Another example is prediction markets. Great example: Microsoft used prediction markets to assess the likely release date of an internal product. Quickly found that their expected release date was way earlier than the people participating thought, and the product manager was then alerted to problems with the project that were known among some of the team, but had not been communicated.
  4. Individual Decision: ‘decide’ tasks where the pieces are independent. For example:
    • the market pattern, where everyone makes individual decisions about when to buy things, at what price. The Amazon Mechanical Turk is another example. One of Tom’s students has written a toolkit for iteratively launching and then intergrating mechanical turk tasks.
    • social network pattern – people make individual decisions, but without any money changing hands. For example the amazon recommendation system.

Observations of this analysis: Genes (patterns) don’t occur in isolation, but in particular combinations. For example, across the range of tasks involved in deciding which wikipedia articles to keep, and editing those articles, many of the different patterns across all four quadrants are used. There are also families of similar combinations. E.g.innocentive and threadless are almost identical in terms of the patterns they use, with the only difference being the threadless also includes a crowd vote.

Tom finished with some speculative comments about seeing us at some point in the future as a single global brain, and closed with a quote from Kevin Kelly’s We are the Web:

There is only one time in the history of each planet when its inhabitants first wire up its innumerable parts to make one large Machine. Later that Machine may run faster, but there is only one time when it is born.

You and I are alive at this moment.

PS: most of the ideas in the talk are in the paper Harnessing crowds.

Usable Climate Science

27. October 2009 · 2 comments · Categories: collaborative science, sustainability

I’ve been invited to give a talk to the Toronto HCI chapter as part of World Usability Day, for which the theme is designing for a sustainable world. Here’s what I have come up with as an abstract for my talk, to be entitled “Usable Climate Science”:

Sustainability is usually defined as “the ability to meet present needs without compromising the ability of future generations to meet their needs”. The current interest in sustainability derives partly from a general concern about environmental degradation and resource depletion, and partly from an awareness of the threat of climate change. But to many people, climate change is only a vague problem, and to some people (e.g. about half the the US population) it isn’t regarded as a problem at all. There is a widespread lack of understanding of the core scientific results of climate science, and the methodology by which those results are obtained. Which in turn means that the public discourse is dominated by ignorance, polarization, and political point scoring. In this environment, lobbyists can propagate misinformation on behalf of various vested interests, and people decide what to believe based on their political worldviews, rather than what the scientific evidence actually says. The chances of getting sound, effective policy in such an environment are slim. In this talk, I will argue that we cannot properly address the challenge of climate change unless this situation is fixed. Furthermore, I’ll argue that the core problem is a usability challenge: how do we make the science itself accessible to the general public? The numerical simulations of climate developed by climatologists are usable only by people with PhDs in climatology. The infographics used to explain climate change in the popular press tend to be high design and low information. What is missing is a concerted attempt to get the core science across to a general audience using software tools and visualizations in which usability is the primary design principle. In short, how do we make climate science usable? Unless we do this, journalists, politicians and the public will be unable to judge whether proposed policy solutions are viable, and unable to distinguish sound science from misinformation. I will illustrate the talk with some suggestions of how we might meet this goal.

Update: talk details have now been announced. It’s on Nov 12 at 7:15pm, in BA1220.

100th Post

25. October 2009 · 1 comment · Categories: blogging

I’ve reached an interesting milestone – this is my 100th post on the blog. When I started the blog in March, I never expected to be able to post regularly. And there have been patches of very few posts (not much in the last month I’m afraid, as I’ve been teaching and writing research proposals). But overall, by treating the blog as part of my research, I’ve managed to blog far more often than I expected.

Someone recently suggested my decision to start blogging was an experiment in constructivist learning. I think that’s accurate, and certainly reflects my own philosophy about learning (and, serendipitously, the google search for a good link on constructivist learning gave me an opportunity to refresh my memory about Vygotsky’s Zone of Proximal Development and Kelly’s Personal Construct Theory, both of which I learned about many years ago when I was at Sussex). Clearly, I need to write a longer post on this, as ZPD is a useful concept to guide communication about climate change – understanding what each audience is ready to grasp is important for ensuring we get through. But anyway…

By becoming a blogger, I’ve learned many things. One is that there’s a huge gulf of incomprehension between the vast majority of (academic) researchers who don’t blog, and those who do. I was struck by this at a conference last week. I arrived a few minutes late to the opening keynote address, which was by Fred Brooks. I opened my laptop, and started taking notes, thinking I would liveblog the talk. Within a few minutes, the conference chair came to me and asked me to close my laptop. It seems that before the talk started, Fred had asked everyone to close their laptops so they could pay attention to the talk, and the conference chair felt that Fred is a sufficiently distinguished that his suggestion should be enforced rigorously. So, having recently gotten used to meetings where a significant segment of the audience is blogging, twittering and friendfeeding as the talk progresses, here was a conference where none of these things were even permissible. I was stunned – this was old-fashioned stodgy academic conferencing at its worst.

One of the things that I love about liveblogging a keynote talk is that it forces me to pay careful attention to the talk. I’m generally taking notes, crosschecking web references, and trying to distill the essence of the talk in realtime, and it’s exhilarating. Without that activity to focus me, I just daydream. So I missed large chunks of Fred Brooks’ talk. I think he said some interesting things about coherence in the design process, but I missed a lot. I thought about taking handwritten notes, but decided  that was useless as (based on past experience) I’d never look at them again. And anyway, the simultaneous googling of ideas in the talk is much of the fun. Liveblogging has spoiled me. And the non-bloggers probably have no idea what I’m talking about.

Something else: before I started blogging, I speculated that most academics don’t blog because they already have a strong online presence, as represented by their papers, books, webpages, etc. I guess I thought that blogging just a way for validating your existence if you don’t already have a body of published work. But as I blog, and come to know other people through their blogs, I realise that it’s a much better way to communicate research. Published papers represent only a fraction of the ideas a researcher is working on – and usually by the time the publication appears, they’re old ideas. The blog gets across the current set of ideas a person is working on. It offers a much better awareness of what people are up to, and is great for starting conversations at conferences (you don’t have to go through the whole “what are you working on these days?” process). So now I think the reason most academics don’t blog is because they’ve been trained not to by an academic system that places value only on peer-reviewed papers, where anything else is seen as a distraction.

Which brings me to one more point: I had naively thought that my blog posts would form the basis of papers I would write, and that I could construct a paper by stringing together several existing posts. Well that never happened, and other people have confirmed that it doesn’t work that way for them either. So I’ve ended up writing fewer papers, but feel I’m getting much more research done. Which will be a very uncomfortable situation for many academics, especially those without tenure yet. But I’m a tenured professor, and quite frankly, I don’t give a damn. There are already too many published papers out there anyway.

Computational Complexity and Climate Change?

06. October 2009 · 1 comment · Categories: climate modeling, research ideas

I attended a talk this morning by Holger Hoos, from UBC, and then had a fascinating conversation with him over lunch. He’s on an 8 week driving tour across Canada and the US, stopping off at universities along the way to meet with colleagues give talks. Great idea – more academics should do this (although I can’t figure out what I’d do with the kids…)

Anyway, what piqued my interest was the framing Holger used for the talk: we live in interesting times, and are faced with many grand challenges: climate change, peak oil, complex diseases, market turmoil, etc. Many of these challenges are due to complexity of various kinds, and to tame this complexity we need to be able to understand, model and control complex systems. And of course, taming complexity is what much of computer science is about.

The core of his talk was a fascinating look at new heuristic algorithms for solving NP hard problems, e.g. algorithms that that outperform the best TSP algorithms and SAT solvers, by using machine learning techniques to tweak the parameters on the heuristics to optimize them for different kinds of input. Which leads to a whole new approach of empirical algorithm design and algorithm engineering. One theme throughout the talk was shift in focus for algorithm design from thinking about worst case analysis, to thinking about handling typical cases, which is something I’ve long felt is a problem with theoretical computer science, and one of the reasons the field has been largely irrelevant when tackling most real engineering problems.

Anyway, for all that I enjoyed the talk, there seemed  to be a gap between the framing (tackling the grand challenges of our time) and the technical content (solutions to computationally intractable problems). Over lunch we talked about this. My observation is that, for climate change in particular, I don’t believe there are any aspects of the challenge that require solving computationally complex problems. It would be nice if there were – it would help me complete my map of how the various subfields of computer science can contribute to tackling climate change. There are obvious applications for information systems (aka databases), graphics and visualization, human computer interaction (usable climate models!!), software engineering, ubiquitous computing (e.g. sensor networks), systems (e.g. power aware computing), and so on.

We talked a little about whether climate models themselves count, but here the main challenges are in optimizing continuous mathematics routines for high performance, rather than solving complex discrete mathematics problems. For example, we speculated whether some of Holger’s work on applying machine learning techniques to parameter tuning could be applied to the parameter schemes for climate models, but even here, I’m not convinced, because there is no oracle. The problem is that climate scientists can’t write down good correctness criteria for climate models because the problem isn’t to develop a “formally correct” model, but rather a scientifically useful one. The model is good if it helps test a scientific hypothesis about how (some aspect of) earth systems work. A model that gets a good fit with observational data because the parameters have been ‘over-tuned’ will get a poor reception in the climate science community; the challenge is to get a model that matches observational data because we’ve correctly understood the underlying physical processes, not because we’ve blindly twiddled the knobs. However, I might be being overly pessimistic about this, and there might be scope for some of these techniques because model tuning still remains a challenging task in climate modeling.

But the more urgent and challenging problems in climate change remain squarely in the realm of how to wean the world off its addiction to fossil fuels as rapidly as possible. This is a problem of information (and overcoming disinformation), of behaviour (individual and social), of economics (although most of modern economic theory is useless in this respect), and of politics. Computer Science has a lot to offer in tackling the information problems, and also some useful abstraction and modeling techniques to understand the other problems. And of course, software is a critical enabling technology in the switch to alternative energy sources. But I still don’t see any computational complexity problems that need solving in all of this. Tell me I’m wrong!

Climate Change Informatics – a proposal

06. October 2009 · 4 comments · Categories: collaborative science, research ideas

Here’s the intro to a draft proposal I’m working on to set up a new research initiative in climate change informatics at U of T (see also: possible participants and ideas for a research agenda). Comments welcome.

Climate change is likely to be the defining issue of the 21st Century. The impacts of a climate change include a dramatic reduction of food production and water supplies, more extreme weather events, the spread of disease, sea level rise, ocean acidification, and mass extinctions. We are faced with the twin challenges of mitigation (avoiding the worst climate change effects by rapidly transitioning the world to a low-carbon economy) and adaptation (re-engineering the infrastructure of modern society so that we can survive and flourish on a hotter planet)
These challenges are global in nature, and pervade all aspects of society. To address them, researchers, engineers, policymakers, and educators from many different disciplines need to come to the table and ask what they can contribute. There are both short-term challenges (such as how to deploy, as rapidly as possible, existing technology to produce renewable energy; how to design government policies and international treaties to bring greenhouse gas emissions under control) and long-term challenges (such as how to complete the transition to a global carbon-neutral society by the latter half of this century).
For Ontario, climate change is both a challenge and an opportunity. The challenge comes in understanding the impacts and adapting to rapid changes in public health, agriculture, management of water and energy resources, transportation, urban planning, and so on. The opportunity is the creation of green jobs through the rapid development of new alternative energy sources and energy conservation measures. Indeed, it is the opportunity to become a world leader in low-carbon technologies.
While many of these challenges and opportunities are already well understood, the role of digital media as both a critical enabling technology and a growing service industry is less well understood. Digital media is critical to effective decision making on climate change issues at all levels. For governmental planning, simulations and visualizations are essential tools for designing and communicating policy choices. For corporations large and small, effective data gathering and business intelligence tools are needed to enable a transition to low-carbon energy solutions. For communities, social networking and web 2.0 technologies are the key tools in bringing people together and enabling coordinated action, and tracking the effectiveness of that action.
Research on climate change has generally clustered around a number of research questions, each studied in isolation. In the physical sciences, the focus is on the physical processes in the atmosphere and biosphere that lead to climate change. In geography and environmental sciences, there is a strong focus on impacts and adaptation. In economics there is a focus on the trade-offs around various policy instruments. In various fields of engineering there is a push for development and deployment of new low-carbon technologies.
Yet climate change is a systemic problem, and effective action requires an inter-disciplinary approach and a clear understanding of how these various spheres of activity interact. We need the appropriate digital infrastructure for these diverse disciplines to share data and results. We need to understand better how social and psychological processes (human behaviour, peer pressure, the media, etc) interact with political processes (policymaking, leadership, voting patterns, etc), and how both are affected by our level of understanding of the physical processes of climate change. And we need to understand how information about all these processes can be factored into effective decision-making.
To address this challenge, we propose the creation of a major new initiative on Climate Change Informatics at the University of Toronto. This will build on existing work across the university on digital media and climate change, and act as a focus for inter-disciplinary research. We will investigate the use of digital media to bridge the gaps between scientific disciplines, policymakers, the media, and public opinion.

Climate change is likely to be the defining issue of the 21st Century. The impacts of a climate change include a dramatic reduction of food production and water supplies, more extreme weather events, the spread of disease, sea level rise, ocean acidification, and mass extinctions. We are faced with the twin challenges of mitigation (avoiding the worst climate change effects by rapidly transitioning the world to a low-carbon economy) and adaptation (re-engineering the infrastructure of modern society so that we can survive and flourish on a hotter planet)

These challenges are global in nature, and pervade all aspects of society. To address them, researchers, engineers, policymakers, and educators from many different disciplines need to come to the table and ask what they can contribute. There are both short-term challenges (such as how to deploy, as rapidly as possible, existing technology to produce renewable energy; how to design government policies and international treaties to bring greenhouse gas emissions under control) and long-term challenges (such as how to complete the transition to a global carbon-neutral society by the latter half of this century).

For Ontario, climate change is both a challenge and an opportunity. The challenge comes in understanding the impacts and adapting to rapid changes in public health, agriculture, management of water and energy resources, transportation, urban planning, and so on. The opportunity is the creation of green jobs through the rapid development of new alternative energy sources and energy conservation measures. Indeed, it is the opportunity to become a world leader in low-carbon technologies.

While many of these challenges and opportunities are already well understood, the role of digital media as both a critical enabling technology and a growing service industry is less well understood. Digital media is critical to effective decision making on climate change issues at all levels. For governmental planning, simulations and visualizations are essential tools for designing and communicating policy choices. For corporations large and small, effective data gathering and business intelligence tools are needed to enable a transition to low-carbon energy solutions. For communities, social networking and web 2.0 technologies are the key tools in bringing people together and enabling coordinated action, and tracking the effectiveness of that action.

Research on climate change has generally clustered around a number of research questions, each studied in isolation. In the physical sciences, the focus is on the physical processes in the atmosphere and biosphere that lead to climate change. In geography and environmental sciences, there is a strong focus on impacts and adaptation. In economics there is a focus on the trade-offs around various policy instruments. In various fields of engineering there is a push for development and deployment of new low-carbon technologies.

Yet climate change is a systemic problem, and effective action requires an inter-disciplinary approach and a clear understanding of how these various spheres of activity interact. We need the appropriate digital infrastructure for these diverse disciplines to share data and results. We need to understand better how social and psychological processes (human behaviour, peer pressure, the media, etc) interact with political processes (policymaking, leadership, voting patterns, etc), and how both are affected by our level of understanding of the physical processes of climate change. And we need to understand how information about all these processes can be factored into effective decision-making.

To address this challenge, we propose the creation of a major new initiative on Climate Change Informatics at the University of Toronto. This will build on existing work across the university on digital media and climate change, and act as a focus for inter-disciplinary research. We will investigate the use of digital media to bridge the gaps between scientific disciplines, policymakers, the media, and public opinion.

Surveying scientific opinion

02. October 2009 · 1 comment · Categories: climate science, research ideas

Survey studies are hard to do well. I’ve been involved in some myself, and have helped many colleagues to design them, and we nearly always end up with problems when it comes to the data analysis. They are a powerful way of answering base-rate questions (i.e. the frequency or severity of some phenomena) or for exploring subjective opinion (which is, of course, what opinion polls do). But most people who design surveys don’t seem to know what they are doing. My checklist for determining if a survey is the right way to approach a particular research question includes the following:

  • Is it clear exactly what population you are interested in?
  • Is there a way to get a representative sample of that population?
  • Do you have resources to obtain a large enough sample?
  • Is it clear what variables need to be measured?
  • Is it clear how to measure them?

Most research surveys have serious problems getting enough people to respond to ensure the results really are representative, and the people who do respond are likely to be a self-selecting group with particularly strong opinions about the topic. Professional opinion pollsters put a lot of work into adjustments for sampling bias, and still often get it wrong. Researchers rarely have the resources to do this (and almost never repeat a survey, so never have the data to do such adjustments anyway). There are also plenty of ways to screw up on the phrasing of the questions and answer modes, such that you can never be sure people have all understood the questions in the same way, and that the available response modes aren’t biasing their responses. (Kitchenham has a good how-to guide)

ClimateSight recently blogged about a fascinating, unpublished survey of whether climate scientists think the IPCC AR4 is an accurate representation our current understanding of climate science. The authors themselves blog about their efforts to get the survey published here, here and here. Although they acknowledge some weaknesses to do with sampling size and representativeness, they basically think the survey itself is sound. Unfortunately, it’s not. As I commented on ClimateSight’s post, methodologically, this survey is a disaster. Here’s why:

The core problem with the paper is the design of the question and response modes. At the heart of their design is a 7-point Likert scale to measure agreement with the conclusions of the IPCC AR4. But this doesn’t work as a design for many reasons:

1) The IPCC AR4 is a massive document, which a huge number of different observations. Any climate scientist will be able to point to bits that are done better and bits that are done worse. Asking about agreement with it, without spelling out which of its many conclusions you’re asking about is hopeless. When people say they agree or disagree with it, you have no idea which of its many conclusions they are reacting to.

2) The response mode used in the study has a built in bias. If the intent is to measure the degree to which scientists think the IPCC accurately reflects, say, the scale of the global warming problem (whatever that means), then central position on the 7-point scale should be “the IPCC got it right”. In the study, this is point 5 on the scale, which immediately introduces a bias because there are twice as many available response modes available in to the left of this position (“IPCC overstates the problem”) than there are to the right (“IPCC understates the problem”). In other words, the scale itself is biased towards one particular pole.

3) The study authors gave detailed descriptive labels to each position on the scale. Although it’s generally regarded as a good idea to give clear labels to each point on a Likert scale, the idea is that this should help users to understand that the intervals on the scale are to be interpreted as roughly equivalent. The labels need to be very simple. The set of labels in this study end up conflating a whole bunch of different ideas, each of which should be tested with a different question and a separate scale. For example, the labels in include ideas such as:

  • fabrication of the science,
  • false hypotheses,
  • natural variation,
  • validity of models,
  • politically motivated scares,
  • divertion of attention,
  • uncertainties,
  • scientists who know what they’re doing,
  • urgency of action,
  • damage to the environment,

…and so on. Conflating all of these onto a single scale makes analysis impossible, because you don’t know which of the many ideas associated with each response mode each respondent is agreeing or disagreeing with. A good survey instrument would ask about only one of these issues at once.

4) Point 5 on the scale (the one interpreted as agreeing with the IPCC) includes the phrase “the lead scientists know what they are doing”. Yet the survey is sent out to select group that includes many such lead scientists and their immediate colleagues. This form of wording immediately biases this group towards this response, regardless of what they think about the overall IPCC findings. Again, asking specifically about different findings in the IPCC report is much more likely to find out what they really think; this study is likely to mask the range of opinions.

5) And finally, as other people have pointed out, the sampling method is very suspect. Although the authors acknowledge that they didn’t do random sampling, and that this limits the kinds of analysis they can do, it also means that any quantitative summary of the responses is likely to be invalid. There’s plenty of reason to suspect that significant clusters of opinion chose not to participate because they saw the questionnaire (especially given some of the wording) as suspect. Given the context for this questionnaire, within a public discourse where everything gets distorted sooner or later, many climate scientists would quite rationally refuse to participate in any such study. Which means really we have no idea if the distribution shown in the study represents the general opinion of any particular group of scientists at all.

So, it’s not surprising no-one wants to publish it. Not because of any concerns for the impact of its findings, but simply because it’s not a valid scientific study. The only conclusions that can be drawn from this study are existence ones:

  1. there exist some people who think the IPCC underestimated (some unspecified aspect of) climate change;
  2. there exist some people who think the IPCC overestimated (some unspecified aspect of) climate change and
  3. there exist some people who think the IPCC scientists know what they are doing.

The results really say nothing about the relative sizes of these three groups, nor even whether the three groups overlap!

Now, the original research question is very interesting, and worth pursuing. Anyone want to work on a proper scientific survey to answer it?