Experimental climatology for kids

13. January 2010 · 1 comment · Categories: climate modeling, research ideas

I picked up Stephen Schneider’s “Science as a Contact Sport” to read on travel this week. I’m not that far into it yet (it’s been a busy trip), but was struck by a comment in chapter 1 about how he got involved in climate modeling. In the late 1960’s, he was working on his PhD thesis in plasma physics, and (in his words) “knew how to calculate magneto-hydro-dynamic shocks at 20,000 times the speed of sound”, with “one-and-a-half dimensional models of ionized gases” (Okay, I admit it, I have no idea what that means, but it sounds impressive)…

…Anyway, along comes Joe Smagorinsky from Princeton, to give a talk on the challenges of modeling the atmosphere as a three-dimensional fluid flow problem on a rotating sphere, and Schneider is immediately fascinated by both the mathematical challenges and the potential of this as important and useful research. He goes on to talk about the early modeling work and the mis-steps made in the early 1970’s on figuring out whether the global cooling from aerosols would be stronger than the global warming from greenhouse gases, and getting the relative magnitudes wrong by running the model without including the stratosphere. And how global warming denialists today like to repeat the line about “first you predicted global cooling, then you predicted global warming…” without understanding that this is exactly how science proceeds, by trying stuff, making mistakes, and learning from them. Or as Ms. Frizzle would say, “Take chances! Make Mistakes! Get Messy!” (No, Schneider doesn’t mention Magic School Bus in the book. He’s too old for that).

Anyway, I didn’t get much further reading the chapter, because my brain decided to have fun with the evocative phrase “modeling the atmosphere as a three-dimensional fluid flow problem on a rotating sphere”, which is perhaps the most succinct description I’ve heard yet of what a climate model is. And what would happen if Ms. Frizzle got hold of this model and encouraged her kids to “get messy” with it. What would they do?

Let’s assume the kids can run the model, and play around with its settings. Let’s assume that they have some wonderfully evocative ways of viewing the outputs too, such as these incredible animations of precipitation from a model (my favourite is “August“) from NCAR, and where greenhouse gases go after we emit them (okay, the latter was real data, rather than a model, but you get the idea).

What experiments might the kids try with the model? How about:

  1. Stop the rotation of the earth. What happens to the storms? Why? (we’ll continue to ask “why?” for each one…)
  2. Remove the land-masses. What happens to the gulf stream?
  3. Remove the ice at the poles. What happens to polar temperatures? Why? (we’ll switch to a different visualization for this one)
  4. Remove all CO2 from the atmosphere. How much colder is the earth? Why? What happens if you leave it running?
  5. Erupt a whole bunch of volcanoes all at once. What happens? Why? How long does the effect last? Does it depend on how many volcanoes you use?
  6. Remove all human activity (i.e. GHG emissions drop to zero instantly). How long does it take for the greenhouse gases to return to the levels they were at before the industrial revolution? Why?
  7. Change the tilt of the earth’s axis a bit. What happens to seasonal variability? Why? Can you induce an ice age? If so, why?
  8. Move the earth a little closer to the sun. What happens to temperatures? How long do they take to stabilize? Why that long?
  9. Burn all the remaining (estimated) reserves of fossil fuels all at once. What happens to temperatures? Sea levels? Polar ice?
  10. Set up the earth as it was in the last ice age. How much colder are global temperatures? How much colder are the poles? Why the difference? How much colder is it where you live?
  11. Melt all the ice at the poles (by whatever means you can). What happens to the coastlines near where you live? Over the rest of your continent? Which country loses the most land area?
  12. Keep CO2 levels constant at the level they were at in 1900, and run a century-long simulation. What happens to temperatures? Now try keeping aerosols constant at 1900 levels instead. What happens? How do these two results compare to what actually happened?

Now compare your answers with what the rest of the class got. And discuss what we’ve learned. [And finally, for the advanced students – look at the model software code, and point to the bits that are responsible for each outcome… Okay, I’m just kidding about that bit. We’d need literate code for that].

Okay, this seems like a worthwhile project. We’d need to wrap a desktop-runnable model in a simple user interface with the appropriate switches and dials. But is there any model out there that would come anywhere close to being useable in a classroom situation for this kind of exercise?

(feel free to suggest more experiments in the comments…)

Documenting climate models

13. January 2010 · 7 comments · Categories: climate modeling

This week I’m visiting the Max Planck Institute for Meteorology (MPI-M) in Hamburg. I gave my talk yesterday on the Hadley study, and it led to some fascinating discussions about software practices used for model building. One of the topics that came up in the discussion afterwards was how this kind of software development compares with agile software practices, and in particular the reliance on face-to-face communication, rather than documentation. Like many software projects, climate modellers struggle to keep good, up-to-date documentation, but generally feel they should be doing better. The problem of course, is that traditional forms of documentation (e.g. large, stand-alone descriptions of design and implementation details) are expensive to maintain, and of questionable value – the typical experience is that you wade through the documentation and discover that despite all the details, it never quite answers your question. Such documents are often produced in a huge burst of enthusiasm for the first release of the software, but then never touched again through subsequent releases. And as the code in the climate models evolves steadily over decades, the chances of any stand-alone documentation keeping up are remote.

An obvious response is that the code itself should be self-documenting. I’ve looked at a lot of climate model code, and readability is somewhat variable (to put it politely). This could be partially addressed with more attention to coding standards, although it’s not clear how familiar you would have to be with the model already to be able to read the code, even with very good coding standards. Initiatives like Clear Climate Code intend to address this problem, by re-implementing climate tools as open source projects in Python, with a strong focus on making the code as understandable as possible. Michael Tobis and I have speculated recently about how we’d scale up this kind of initiative to the development of coupled GCMs.

But readable code won’t fill the need for a higher level explanation of the physical equations and their numerical approximations used in the model, along with rationale for algorithm choices. These are often written up in various forms of (short) white papers when the numerical routines are first developed, and as these core routines rarely change, this form of documentation tends to remain useful. The problem is that these white papers tend to have no official status (or perhaps at best, they appear as technical reports), and are not linked in any usable way to distributions of the source code. The idea of literate programming was meant to solve this problem, but it never took off, probably because it demands that programmers must tear themselves away from using programming languages as their main form of expression, and start thinking about how to express themselves to other human beings. Given that most programmers define themselves in terms of the programming languages they are fluent in, the tyranny of the source code is unlikely to disappear anytime soon. In this respect, climate modelers have a very different culture from most other kinds of software development teams, so perhaps this is an area where the ideas of literate programming could take root.

Lack of access to these white papers could also be solved by publishing them as journal papers (thus instantly making them citeable objects). However, scientific journals tend not to publish descriptions of the designs of climate models, unless they are accompanied with new scientific results from the models. There are occasional exceptions (e.g. see the special issue of the Journal of Climate devoted to the MPI-M models). But things are changing, with the recent appearance of two new journals:

  • Geoscientific Model Development, an open access journal that accepts technical descriptions of the development and evaluation of the models;
  • Earth Science Informatics, a Springer Journal with a broader remit than GMD, but which does cover descriptions of the development of computational tools for climate science.

The problem is related to another dilemma in climate modeling groups: acknowledgement for the contributions of those who devote themselves more to model development rather than doing “publishable science”. Most of the code development is done by scientists whose performance is assessed by their publication record. Some modeling centres have created job positions such as “programmers” or “systems staff”, although most people hired into these roles have a very strong geosciences background. A growing recognition of the importance of their contributions represents a major culture change in the climate modeling community over the last decade.

Do “Many eyeballs make all bugs shallow”?

11. December 2009 · 9 comments · Categories: climate modeling

As a follow-on from yesterday’s post on making climate software open source, I’d like to pick up on the oft-repeated slogan “Many eyeballs make all bugs shallow”. This is sometimes referred to as Linus’ Law (after Linus Torvalds, creator of Linux), although this phrase is actually attributed to Eric Raymond (Torvalds would prefer “Linus’s Law” to be something completely different). Judging from the number of times this slogan is repeated in the blogosphere, there must be lots of very credulous people out there. (Where are the real skeptics when you need them?)

Robert Glass tears this one apart as a myth in his book “Facts and Fallacies about Software Engineering“, on the basis of three points: it’s self-evidently not true (the depth of a bug has nothing to do with how many people are looking for it); there’s plenty of empirical evidence that the utility of adding additional reviewers to a review team tails off very quickly after around 3-4 reviewers; and finally there is no empirical evidence that open source software is less buggy than its alternatives.

More interestingly, companies like Coverity, who specialize in static analysis tools, love to run their tools over open source software and boast about the number of bugs they find (it shows off what their tools can do). For example, their 2009 study found 38,453 bugs in 60 million lines of source code (a bug density of about 0.64 defect/KLOC). Quite clearly, there are many types of bugs that you need automated tools to find, no matter how many eyeballs have looked at the code.

Part of the problem is that the “many eyeballs” part isn’t actually true anyway. In a study conducted by Xu et. al. in 2005 of the sourceforge community, they found that participation in projects follows the power law well known in social network theory: a few open source projects have a very large number of participants, and a very large number have very few participants. Similarly, a very small number of open source developers participate in lots of projects; the majority participate in just one or two:

SourceForge Project and Developer Community Scale Free Degree Distributions (Figure 7d from Xu et al 2005)

SourceForge Project and Developer Community Scale Free Degree Distributions (Figure 7d from Xu et al 2005)

For example, the data shown in these graphs include all developers and active users for about 160,000 sourceforge projects. Of these projects, 25% had only a single person involved (as either developer or user!), and a further 10% had only 2-3 people involved. Clearly, a significant number of open source projects never manage to build a community of any size.

This is relevant to the climate science community because many of the tens of thousands of scientists actively pursing research relevant to our understanding of climate change build software. If all of them release their software as open source, there’s no reason to expect a different distribution from the graphs above. So most of this software will never attract any participants outside the handful of scientists who wrote it, because there simply aren’t enough eyeballs or interest available. The kind of software described in the famous “Harry” files at the CRU is exactly of this nature – if it hadn’t been picked out in the stolen CRU emails, nobody other than “Harry” would ever take the time to look at it. And even if lots of people’s attention was drawn to this particular software (as it has been), there are still thousands of other scraps of similar software out there which would also remain single person projects like those on sourceforge. In contrast, a very small number of projects will attract hundreds of developers/users.

The thing is, this is exactly how the climate science community operates already. A small number of projects (like the big GCMs, listed here) already have a large number of developers and users – for example, CCSM and Hadley’s UM have hundreds of active developers, and a very mature review process. Meanwhile a very large number of custom data analysis tools are built by a single person for his/her own use. Declaring all of these projects to be open source will not magically bring “many eyeballs” to bear on them. And indeed, as Cameron Neylon argues, those that do will immediately have to protect themselves from a large number of clueless newbies by doing exactly what many successful open source projects do: the inner clique closes ranks and refuses to deal with outsiders, ignores questions on the mailing lists, etc. Isn’t that supposed to be the problem we were trying to solve?

The argument that making climate software open source will somehow magically make it higher quality is therefore specious. The big climate models already have many eyeballs, and the small data handling tools will never attract large numbers of eyeballs. So, if any of the people screaming about openness are truly interested in improving software quality, they’ll argue for something that is actually likely to make a difference.

Software Quality in Climate Research

08. December 2009 · 8 comments · Categories: climate modeling, climate science, software tools

Well, this is what it comes down to. Code reviews on national TV. Who would have thought it? And, by the standards of a Newsnight code review, the code in question doesn’t look so good. Well, it’s not surprising it doesn’t. It’s the work of one, untrained programmer, working in an academic environment, trying to reconstruct someone else’s data analysis. And given the way in which the CRU files were stolen, we can be pretty sure this is not a random sample of code from the CRU; it’s handpicked to be one of the worst examples.

Watch the clip from about 2:00. They compare the code with some NASA code, although we’re not told what exactly. Well, duh. If you compare the experimental code written by one scientist on his own, which has clearly not been through any code review, with that produced by a NASA’s engineering processes, of course it looks messy. For any programmers reading this: How many of you can honestly say that you’d come out looking good if I trawled through your files, picked the worst piece of code lying around in there, and reviewed it on national TV? And the “software engineer” on the program says it’s “below the standards you would expect in any commercial software”. Well, I’ve seen a lot of commercial software. It’s a mix of good, bad, and ugly. If you’re deliberate with your sampling technique, you can find a lot worse out there.

Does any of this matter? Well, a number of things bug me about how this is being presented in the media and blogosphere:

  • The first, obviously, is the ridiculous conclusion that many people seem to be making that poor code quality in one, deliberately selected program file somehow invalidates all of climate science. As cdavid points out towards the end of this discussion, if you’re going to do that, then you pretty much have to throw out most results in every field of science over the past few decades for the same reason. Bad code is endemic in science.
  • The slightly more nuanced, but equally specious, conclusion that bugs in this code mean that research results at the CRU must be wrong. Eric Raymond picks out an example he calls blatant data-cooking, but is quite clearly fishing for results, because he ignores the fact that the correction he picks on is never used in the code, except in parts that are commented out. He’s quote mining for effect, and given Raymond’s political views, it’s not surprising. Just for fun, someone quote mined Raymond’s own code, and was horrified at what he found. Clearly we have to avoid all open source code immediately because of this…? The problem, of course, is that none of these quote miners have gone to the trouble to establish what this particular code is, why it was written, and what it was used for.
  • The widely repeated assertion that this just proves that scientific software must be made open source, so that a broader community of people can review it and improve it.

It’s this last point that bothers me most, because at first sight, it seems very reasonable. But actually, it’s a red herring. To understand why, we need to pick apart two different arguments:

  1. An argument that when a paper is published, all of the code and data on which it is based should be released so that other scientists (who have the appropriate background) can re-run it and validate the results. In fields with complex, messy datasets, this is exceedingly hard, but might be achievable with good tools. The complete toolset needed to do this does not exist today, so just calling for making the code open source is pointless. Much climate code is already open source, but that doesn’t mean anyone in another lab can repeat a run and check the results. The problems of reproducibility have very little to do with whether the code is open – the key problem is to capture the entire scientific workflow and all data provenance. This is very much an active line of research, and we have a long way to go. In the absence of this, we rely on other scientists testing the results with other methods, rather than repeating the same tests. Which is the way it’s done in most branches of science.
  2. An argument that there is a big community of open source programmers out there who could help. This is based on a fundamental misconception about why open source software development works. It matters how the community is organised, and how contributions to the code are controlled by a small group of experts. It matters that it works as a meritocracy, where programmers need to prove their ability before they are accepted into the inner developer group. And most of all, it matters that the developers are the domain experts. For example, the developers who built the Linux kernel are world-class experts on operating systems and computer architecture. Quite often they don’t realize just how high their level of expertise is, because they hang out with others who also have the same level of expertise. Likewise, it takes years of training to understand the dynamics of atmospheric physics in order to be able to contribute to the development of a climate simulation model. There is not a big pool of people with the appropriate expertise to contribute to open source climate model development, and nor is there ever likely to be, unless we expand our PhD programs in climatology dramatically (I’m sure the nay-sayers would like that!).

We do know that most of the heavy duty climate models are built at large government research centres, rather than at universities. Dave Randall explains why this is: the operational overhead of developing, testing and maintaining a Global Climate Model is far too high for university-based researchers. The Universities use (parts of) the models, and do further data analysis on both observational data and outputs from the big models. Much of this is the work of indivdual PhD students or postdocs. Which means that the argument that all code written at all stages of climate research must meet some gold standard of code quality is about as sensible as saying no programmer should ever be allowed to throw together a script to test out if some idea works. Of course bad code will get written in a hurry. What matters is that as a particular line of research matures, the coding practices associated with it should mature too. And we have plenty of evidence that this is true of climate science: the software practices used at the Hadley Centre for their climate models are better than most commercial software practices. Furthermore, they manage to produce code that appears to be less buggy than just about any other code anywhere (although we’re still trying to validate this result, and understand what it means).

None of this excuses bad code written by scientists. But the sensible response to this problem is to figure out how to train scientists to be better programmers, rather than argue that some community of programmers other than scientists can take on the job instead. The idea of open source climate software is great, but it won’t magically make the code better.

Computational Complexity and Climate Change?

06. October 2009 · 1 comment · Categories: climate modeling, research ideas

I attended a talk this morning by Holger Hoos, from UBC, and then had a fascinating conversation with him over lunch. He’s on an 8 week driving tour across Canada and the US, stopping off at universities along the way to meet with colleagues give talks. Great idea – more academics should do this (although I can’t figure out what I’d do with the kids…)

Anyway, what piqued my interest was the framing Holger used for the talk: we live in interesting times, and are faced with many grand challenges: climate change, peak oil, complex diseases, market turmoil, etc. Many of these challenges are due to complexity of various kinds, and to tame this complexity we need to be able to understand, model and control complex systems. And of course, taming complexity is what much of computer science is about.

The core of his talk was a fascinating look at new heuristic algorithms for solving NP hard problems, e.g. algorithms that that outperform the best TSP algorithms and SAT solvers, by using machine learning techniques to tweak the parameters on the heuristics to optimize them for different kinds of input. Which leads to a whole new approach of empirical algorithm design and algorithm engineering. One theme throughout the talk was shift in focus for algorithm design from thinking about worst case analysis, to thinking about handling typical cases, which is something I’ve long felt is a problem with theoretical computer science, and one of the reasons the field has been largely irrelevant when tackling most real engineering problems.

Anyway, for all that I enjoyed the talk, there seemed  to be a gap between the framing (tackling the grand challenges of our time) and the technical content (solutions to computationally intractable problems). Over lunch we talked about this. My observation is that, for climate change in particular, I don’t believe there are any aspects of the challenge that require solving computationally complex problems. It would be nice if there were – it would help me complete my map of how the various subfields of computer science can contribute to tackling climate change. There are obvious applications for information systems (aka databases), graphics and visualization, human computer interaction (usable climate models!!), software engineering, ubiquitous computing (e.g. sensor networks), systems (e.g. power aware computing), and so on.

We talked a little about whether climate models themselves count, but here the main challenges are in optimizing continuous mathematics routines for high performance, rather than solving complex discrete mathematics problems. For example, we speculated whether some of Holger’s work on applying machine learning techniques to parameter tuning could be applied to the parameter schemes for climate models, but even here, I’m not convinced, because there is no oracle. The problem is that climate scientists can’t write down good correctness criteria for climate models because the problem isn’t to develop a “formally correct” model, but rather a scientifically useful one. The model is good if it helps test a scientific hypothesis about how (some aspect of) earth systems work. A model that gets a good fit with observational data because the parameters have been ‘over-tuned’ will get a poor reception in the climate science community; the challenge is to get a model that matches observational data because we’ve correctly understood the underlying physical processes, not because we’ve blindly twiddled the knobs. However, I might be being overly pessimistic about this, and there might be scope for some of these techniques because model tuning still remains a challenging task in climate modeling.

But the more urgent and challenging problems in climate change remain squarely in the realm of how to wean the world off its addiction to fossil fuels as rapidly as possible. This is a problem of information (and overcoming disinformation), of behaviour (individual and social), of economics (although most of modern economic theory is useless in this respect), and of politics. Computer Science has a lot to offer in tackling the information problems, and also some useful abstraction and modeling techniques to understand the other problems. And of course, software is a critical enabling technology in the switch to alternative energy sources. But I still don’t see any computational complexity problems that need solving in all of this. Tell me I’m wrong!

How do people decide which model to use?

20. July 2009 · Write a comment · Categories: climate modeling, climate science

In our climate brainstorming session last week, we invited two postdocs (Chris and Lawrence) from the atmospheric physics group to come and talk to us about their experiences of using climate models. Most of the discussion focussed on which models they use and why, and what problems they experience. They’ve been using the GFDL models, specifically AM2 and AM3 (atmosphere only models) for most of their research, largely because of legacy: they’re working with Paul Kushner, who is from GFDL,  and the group now has many years experience working with these models. However, they’re now faced with having to switch to NCAR’s Community Climate System Model (CCSM). Why? Because the university has acquired a new IBM supercomputer, and the GFDL models won’t run on it (without a large effort to port them). The resulting dilemma reveals a lot about the current state of climate model engineering:

  • If they stick with the GFDL models, they can’t make use of the new supercomputer, hence miss out on a great opportunity to accelerate their research (they could do many more model runs).
  • If they switch to CCSM, they lose a large investment in understanding and working with the GFDL models. This includes both their knowledge of the model (some of their research involves making changes to the code to explore how perturbations affect the runs), and the investment in tools and scripts for dealing with model outputs, diagnostics, etc.

Of course, the obvious solution would be to port the GFDL models to the new IBM hardware. But this turns out to be hard because the models were never designed for portability. Right now, the GFDL models won’t even compile on the IBM compiler, because of differences in how picky different compilers are over syntax and style checking – climate models tend to have many coding idiosyncrasies that are never fixed because the usual compiler never complains about them: e.g. see Jon’s analysis of static checking NASA’s modelE. And even if they fix all these and get the compiler to accept the code, they’re still faced with extensive testing to make sure the models’ runtime behaviour is correct on the new hardware.

There’s also a big difference in support available. GFDL doesn’t have the resources to support external users (particularly ambitious attempts to port the code). In contrast, NCAR has extensive support for the CCSM, because they have made community building an explicit goal. Hence, CCSM is much more like an open source project. Which sounds great, but it also comes at a cost. NCAR have to devote significant resources to supporting the community. And making the model open and flexible (for use by a broader community) hampers their ability to get the latest science into the model quickly. Which leads me to hypothesize that it is the diversity of your user-base that most restricts the ongoing rate of evolution of a software system. For a climate modeling centers like GFDL, if you don’t have to worry about developing for multiple platforms and diverse users, you can get new ideas into the model much quicker.

Which brings me to a similar discussion over the choice of weather prediction models in the UK. Bryan recently posted an article about the choice between WRF (NCAR’s mesoscale weather model) versus the UM (the UK Met Office’s model). Alan posted a lengthy response which echoes much of what I said above (but with much more detail): basically the WRF is well supported and flexible for a diverse community. The UM has many advantages (particularly speed), but is basically unsupported outside the Met Office. He concludes that someone should re-write the UM to run on other hardware (specifically massively parallel machines), and presumably set up the kind of community support that NCAR has. But funding for this seems unlikely.

Designing your own Earth System Model

15. July 2009 · 8 comments · Categories: climate modeling, courses

Spurred on by Michael Tobis’s thoughts about building a readable/understandable climate model in Python, I dug up an old bookmark to a course offered in UMinn’s geology department on Designing your own “Earth System” Model. Really neat idea for a course. Now, with a bit of tweaking, I could set up a similar course here, but with a twist – we recruit a mix of CS and Physics students onto the course, and put them together in cross-disciplinary teams, each team building an earth system model of some kind (tapping into the domain knowledge of the physicists), using current CS and software engineering tools and techniques. And to keep Michael happy, they code it all in Python. Marks awarded based on understandability of the code.

In the longer term, we keep the models produced from each instance of the course, and get the next cohort of students to develop them further – the aim is to build them up to be full scale earth system models.

Update: Michael has posted some more reflections on this.

Growth of code complexity

02. July 2009 · 2 comments · Categories: climate modeling

I’m busy revising our paper on the study of the software development processes at the Hadley Centre for publication in CiSE (yay!). And I just looked again at the graph of code growth (click for bigger version):

UM_evolution2

The top line (green) shows lines of code, while the bottom line (blue) shows number of files. When I first produced this figure last summer, I was struck by the almost linear growth in lines of code over the fifteen years (with two obvious hiccups when core modules were replaced). Other studies have shown that lines of code is a good proxy for functionality, so I interpret this as a steady growth in functionality. Which contrasts with Lehman’s observations that for industrial software, an inverse square curve offers the best fit. Lehman offers as an explanation the theory that growing complexity of the software inevitably slows the addition of new functionality. He claims that his pattern is robust in other (commercial) software he has studied (but I’d haven’t trawled through his papers to see if he gives more case studies).

Subsequently, Godfrey & Tu showed that the Linux kernel did not suffer this limitation, but instead grew slightly faster than linearly (or geometrically if you include the device drivers, which you probably shouldn’t). So, that’s two studies that break Lehman’s pattern: the Linux kernel and the Hadley UM. What do they have in common that’s different from the commercial software systems that Lehman studied? I hypothesize the most likely explanation is that in both cases the code is written by the most knowledgeable domain experts, working in a non-hierarchical meritocracy (by which I mean that no one tells them what to work on, and that they get accepted as members of the development team by demonstrating their ability over a period of time). This isn’t a new hypothesis: Dewayne Perry has been saying for ages that domain expertise by the developers is the single biggest factor in project success.

Anyway, my co-author, Tim, was struck quite a different observation about the graph: the way the two lines diverged over the fifteen years shown. While lines of code have grown relatively fast (nearly tenfold over the fifteen years shown), the number of files has grown much more slowly (only threefold). Which means the average filesize has steadily grown too. What does this mean? Roughly speaking, new files mean the addition of new modules, while new lines within an existing file mean additional functionality within existing modules (although this is not quite correct, as scientific programmers don’t always use separate files for separate architectural modules). Which means more of the growth comes from adding complexity within the existing routines, rather than from expanding the model’s scope. I’m willing to bet a lot of that intra-file growth comes from adding lots of different options for different model configurations.

Getting the source code for climate models

24. June 2009 · 9 comments · Categories: climate modeling, climate science

First, we have to be clear what we mean by a climate model. Wikipedia offers a quick intro to types of climate model. For example:

  • zero dimension models, essentially just a set of equations for the earth’s radiation balance
  • 1-dimensional models – for example where you take latitude into account, as the angle of the sun’s rays matter)
  • EMICS – earth-system models of intermediate complexity
  • GCMs – General Circulation Models (a.k.a Global Climate Models), which model the atmosphere in four dimensions (3D+time), by dividing it into a grid of cubes, and solving the equations of fluid motion for each cube at each time step. While the core of a GCM is usually the atmosphere model, GCMs can be coupled to three dimensional ocean models, or run uncoupled, so that you can have A-GCMs (atmosphere only), and AO-GCMs (atmosphere and ocean). Ocean models are just called ocean models 🙂
  • Earth System Models – Take a GCM, and couple it to models of other earth system processes: sea ice, land ice, atmospheric chemistry, the carbon cycle, human activities such as energy consumption and economics, and so on.

Current research tends to focus on Earth System Models, but for the last round of the IPCC assessment, AO-GCMs were used to generate most of the forecast runs. Here are the 23 AO-GCMs used in the IPCC AR4 assessment, with whatever info I could find about availability of each model :

Now, if you were paying attention, you’ll have noticed that that wasn’t 23 bullet points. Some labs contributed runs from more than one version of their model(s), so it does add up somehow.

Short summary: easiest source code to access: (1) IPSL (includes Trac access!), (2) CCSM and (3) ModelE.

Future work: take a look at the additional models that took part in the Coupled Model Inter-comparison Project (CMIP-3), and see if any of them are also available.

Update: RealClimate has started compiling a fuller list of available codes and datasets.

Agent-based simulations

14. April 2009 · Write a comment · Categories: climate modeling, climate science, systems thinking · Tags: , , , , , , , , ,

Had an interesting conversation this afternoon with Brad Bass. Brad is a prof in the Centre for Environment at U of T, and was one of the pioneers of the use of models to explore adaptations to climate change. His agent based simulations explore how systems react to environmental change, e.g. exploring population balance among animals, insects, the growth of vector-borne diseases, and even entire cities. One of his models is Cobweb, an open-source platform for agent-based simulations. 

He’s also involved in the Canadian Climate Change Scenarios Network, which takes outputs from the major climate simulation models around the world, and extracts information on the regional effects on Canada, particularly relevant for scientists who want to know about variability and extremes on a regional scale.

We also talked a lot about educating kids, and kicked around some ideas for how you could give kids simplified simulation models to play with (along the line that Jon was exploring as a possible project), to get them doing hands on experimentation with the effects of climate change. We might get one of our summer students to explore this idea, and Brad has promised to come talk to them in May once they start with us.

Oh, and Brad is also an expert on green roofs, and will be demonstrating them to grade 5 kids at the Kids World of Energy Festival.

Modularizing Climate Models

31. March 2009 · 1 comment · Categories: climate modeling, climate science, research ideas · Tags: , ,

One of the things that came up in our weekly brainstorming session today was the question of whether climate models can be made more modular, to permit distributed development, and distributed execution. Carolyn has already blogged about some of these ideas. Here’s a little bit of history for this topic.

First, a very old (well, 1989) paper by Kalnay et al,  on Data Interchange Formats, in which they float the idea of “plug compatibility” for climate model components. For a long time, this idea seems to have been accepted as the long term goal for the architecture for climate models. But no-one appears to have come close. In 1996, David Randall wrote an interesting introspective on how university teams can (or can’t) participate in climate model building, in which he speculates that plug compatibility might not be achievable in practice because of the complexity of the physical processes being simulated, and the complex interactions between them. He also points out that all climate models (up to that point) had each been developed at a single site, and he talks a bit about why this appears to be necessarily so.

Fast forward to a paper by Dickinson et al in 2002, which summarizes the results of a series of workshops on how to develop a better software infrastructure for model sharing, and talks about some prototype software frameworks. Then, a paper by Larson et al in 2004, introducing a common component architecture for earth system models, and a bit about the Earth System Modeling Framework being developed at NCAR. And finally, Drake et al.’s Overview of the Community Climate System Model, which appears to use these frameworks very successfully.

Now, admittedly I haven’t looked closely at the CCSM. But I have looked closely at the Met Office’s Unified Model and the Canadian CCCma, and neither of them get anywhere close to the ideal of modularity. In both cases, the developers have to invest months of effort to ‘naturalize’ code contributed from other labs, in the manner described in Randall’s paper.

So, here’s the mystery. Has the CCSM really achieved the modularity that others are only dreaming of? And if so how? The key test would be how much effort it takes to ‘plug in’ a module developed elsewhere…

Great women in climate science

23. March 2009 · 1 comment · Categories: climate modeling, climate science · Tags: , ,

In honour of Ada Lovelace day, I decided to write a post today about Prof Julia Slingo, the new chief scientist at the UK Met Office. News of Julia’s appointment came out in the summer last year during my visit to the Met Office, coincidentally on the same day that I met her, at a workshop on the HiGEM project (where, incidentally, I saw some very cool simulations of ocean temperatures). Julia’s role at the meeting was to represent the sponsor (NERC – the UK equivalent of Canada’s NSERC), but what impressed me about her talk was both her detailed knowledge of the project, and the way she nurtured it – she’ll make a great chief scientist.

Julia’s research has focussed on tropical variability, particularly improving our understanding of the monsoons, but she’s also played a key role in earth system modeling, and especially in the exploration of high resolution models. But best of all, she’s just published a very readable account of the challenges in developing the next generation of climate models. Highly recommended for a good introduction to the state of the art in climate modeling.

A Requirements Modeling Challenge

21. March 2009 · 1 comment · Categories: climate modeling, data modeling, research ideas · Tags: , , ,

Here’s a challenge for the requirements modelling experts. I’ve phrased it as an exam question for my graduate course on requirements engineering (the course is on hiatus, which is lucky, because it would be a long exam…):

Q: The governments of all the nations on a small blue planet want to fix a problem with the way their reliance on fossil fuels is altering the planet’s climate. Draw a goal model (using any appropriate goal modeling notation) showing the key stakeholders, their interdependencies, and their goals. Be sure to show how the set of solutions they are considering contribute to satisfying their goals. The attached documents may be useful in answering this question: (a) A outline of the top level goals; (b) A description of the available solutions, characterized as a set of Stabilization Wedges; (c) A domain expert’s view of the feasbility of the solutions.

Update: Someone’s done the initial identification of actors already.

Meta-data for climate datasets

18. March 2009 · Write a comment · Categories: climate modeling, data modeling · Tags: , , ,

Developing more descriptive meta-data for the huge datasets used by climate scientists is an area where software engineering and information systems research can make a useful contribution. The goal is to develop rich metadata descriptors, based on an ontology that spans the field: the physical processes that climate scientists simulate, the physical quantities used in their models, the configuration settings used in creating model runs, the scientific questions these runs are intended to investigate, and the contents of the datasets used. This ontology can then be used to unify the metadata for model configurations and datasets, so that in principle, the metadata description would be rich enough to allow a researcher to set up a model to reproduce the data in the dataset, and to support rich queries over the datasets (e.g. which model runs isolate the effects of volcanoes on 20th century climate?).

There are two big research projects working on this:

  1. Earth System Curator, which is a US project, and includes as partners NCAR, GFDL, and Georgia Tech. Spencer Rugaber, who is part of the project team, will be coming to ICSE and will talk about Curator in the Software Engineering for the Planet session, and hopefully show off some demos. Here’s a more detailed research paper on Curator.
  2. Metafor, which is a big EU project, and includes as partners the Hadley Centre, IPSL, BADC and others. This project has started to develop some use cases,  and various UML models. No research papers yet, but plenty of information about the project on their website.

understanding the science

16. March 2009 · Write a comment · Categories: climate modeling, climate science, collaborative science, philosophy · Tags: , , ,

I originally wrote this as a response to a post on RealClimate on hypothesis testing

I think one of the major challenges with public understanding of climate change is that most people have no idea of what climate scientists actually do. In the study I did last summer of the software development practices at the Hadley Centre, my original goal was to look just at the “software engineering” of climate simulation models -i.e. how the code is developed and tested. But the more time I spend with climate scientists, the more I’m fascinated by the kind of science they do, and the role of computational models within it.

The most striking observation I have is that climate scientists have a deep understanding of the fact that climate models are only approximations of earth system processes, and that most of their effort is devoted to improving our understanding of these processes (“All models are wrong, but some are useful” – George Box). They also intuitively understand the core ideas from general systems theory – that you can get good models of system-level processes even when many of the sub-systems are poorly understood, as long as you’re smart about choices of which approximations to use. The computational models have an interesting status in this endeavour: they seem to be used primarily for hypothesis testing, rather than for forecasting. A large part of the time, climate scientists are “tinkering” with the models, probing their weaknesses, measuring uncertainty, identifying which components contribute to errors, looking for ways to improve them, etc. But the public generally only sees the bit where the models are used to make long term IPCC-style predictions.

I never saw a scientist doing a single run of a model and comparing it against observations. The simplest use of models is to construct a “controlled experiment” by making a small change to the model (e.g. a potential improvement to how it implements some piece of the physics), comparing this against a control run (typically the previous run without the latest change), and comparing both runs against the observational data. In other words, there is a 3-way comparison: old model vs. new model vs. observational data, where it is explicitly acknowledged that there may be errors in any of the three. I also see more and more effort put into “ensembles” of various kinds: model intercomparison projects, perturbed physics ensembles, varied initial conditions, and so on. In this respect, the science seems to have changed (matured) a lot in the last few years, but that’s hard for me to verify.

It’s a pretty sophisticated science. I would suggest that the general public might be much better served by good explanations of how this science works, rather than with explanations of the physics and mathematics of climate systems.

« Older articles
Newer articles »