Course Policy for use of generative AI

06. January 2024 · Write a comment · Categories: courses · Tags: ,

I’m teaching a new course this term, called Confronting the Climate Crisis. As it’s the first time I’ve taught since the emergence of the latest wave of AI chatbots. Here’s what I came up with:

The assignments on this course have been carefully designed to give you meaningful experiences that build your knowledge and skills, and I hope you will engage with them in that spirit. If you decide to use any AI tools, you *must* include a note explaining what tools you used and how you used them, and include a reflection on how they have affected your learning process. Without such a note, use of AI tools will be treated as an academic offence, with the same penalties as if you had asked someone else (rather than a bot) to do the work for you.

Rationale for this policy: In the last couple of years, so-called Artificial Intelligence (AI) tools have become commonplace, particularly tools that use generative AI to create text and images. The underlying technology uses complex statistical models of typical sequences of words (and elements of images), which can instantly create very plausible responses to a variety of prompts. However, these tools have no understanding of the meanings that we humans attach to words and images, and no experience of the world in which those meanings reside. The result is that they are expert at mimicking how humans express themselves, but they are often factually wrong, and their outputs reflect the biases (racial, gender, socio-economic, geographic) that are inherent in the data on which the models were trained. If you choose to use AI tools to help you create your assignments for this course, you will still be responsible for any inaccuracies and biases in the generated content.

More importantly, these AI tools raise important questions about the nature of learning in higher education. Unfortunately, we have built a higher education system that places far too much emphasis on deadlines and grades, rather than on learning and reflection. In short, we have built a system that encourages students to cheat. The AI industry promotes its products as helpful tools, perhaps no different from using a calculator in math, or a word processor when writing. And there are senses in which this is true – for example if you suffer from writer’s block, an AI tool can quickly generate an outline or a first draft to get you started. But the crucial factor in deciding when and how to use such tools is a question of what, exactly, you are offloading onto the machine. If a tool helps you overcome some of the tedious, low-level steps so that you can move on faster to the important learning experiences, that’s great! If on the other hand, the tool does all the work for you, so you never have to think or reflect on the course material, you will gain very little from this course other than (perhaps) a good grade. In that sense, most of the ways you might use an AI tool in your coursework are no different from other forms of ‘cheating’: they provide a shortcut to a good grade, by skipping the learning process you would experience if you did the work yourself.

Icon for Creative Commons licence CC-BY-NC-SA

This course policy is licensed under a Creative Commons Licence CC BY-NC-SA 4.0. Feel free to use and adapt for non-commercial purposes, as long as you credit me, and share alike any adaptations you make.

Does your model ever surprise you?

20. May 2009 · Write a comment · Categories: ICSE 2009, research ideas · Tags: , , , ,

So, here’s an interesting thought that came up the the Michael Jackson festschrift yesterday. Michael commented in his talk that understanding is not a state, it’s a process. David Notkin then asked how we can know how well we’re doing in that process. I suggested that one of the ways you know is by discovering where your understanding is incorrect, which can happen if your model surprises you. I noticed this is a basic mode of operation for earth system modelers. They put their current best understanding of the various earth systems (atmosphere, ocean, carbon cycle, atmospheric chemistry, soil hydrology, etc) into a coupled simulation model and run it. Whenever the model surprises them, they know they’re probing the limits of their understanding. For example, the current generation of models at the Hadley centre don’t get the Indian Monsoon in the right place at the right time. So they know there’s something in that part of the model they don’t yet understand sufficiently.

Contrast this with the way we use (and teach) modeling in software engineering. For example, students construct UML models as part of a course in requirements analysis. They hand in their models, and we grade them. But at no point in the process do the models ever surprise their authors. UML models don’t appear to have the capacity for surprise. Which is unfortunate, given what the students did in previous courses. In their programming courses, they were constantly surprised. Their programs didn’t compile. Then they didn’t run. Then they kept crashing. Then they gave the wrong outputs. At every point, the surprise is a learning opportunity, because it means there was something wrong with their understanding, which they have to fix. This contrast explains a lot about the relative value students get from programming courses versus software modeling courses.

Now of course, we do have some software engineering modeling frameworks that have the capacity for surprise. They allow you to create a model and play with it, and sometimes get unexpected results. For example, Alloy. And I guess model checkers have that capacity too. A necessary condition is that you can express some property that your model ought to have, and then automatically check that it does have it. But that’s not sufficient, because if the properties you express aren’t particularly interesting, or are trivially satisifed, you still won’t be surprised. For example, UML syntax checkers fall into this category – when your model fails a syntax check, that’s not surprising, it’s just annoying. Also, you don’t necessarily have to formally state the properties – but you do have to at least have clear expectations. When the model doesn’t meet those expectations, you get the surprise. So surprise isn’t just about executability, it’s really about falsifiability.

Liveblogging the EGU – day 4

23. April 2009 · Write a comment · Categories: climate science, EGU 2009 · Tags: , , , , , , ,

8:30: This morning I’m at a session on education. Just my luck – I got here towards the end of the first talk, which looked very interesting. It was Zoe Robinson, from Keele, giving a talk entitled Effective and responsible teaching of climate change in Earth Science-related disciplines. She talked about many of the dilemmas faced in teaching of the earth sciences – e.g. attitudes of students such as climate skepticism, feelings of saturation coverage, etc., coupled with flawed knowledge. At the same time, accurate coverage of the complexity of the earth systems and areas of uncertainty can undermine the sense of urgency and responsibility appropriate for dealing with the challenges of climate change. Zoe talked about non-traditional techniques to use in the classroom to deal with these issues (but I missed most of it!)

8:45: Ilona Pojtok-Tari, Geography Netquipment (project webpage)

9:00: Marina Manea on Online Geodynamics. Talked about how they teach computational geodynamics. using various online modeling tools. The animations on convection in the earth’s mantle look pretty impressive, although the server is pretty slow.

The next two talks picked up a on common theme: how increased time pressure and reduced classroom time are eroding competence in students. Both talked about using other forms of instruction to combat this.

9:15: Stefan Winkler, talking about Geographic-didactical games as interactive tools. They noticed an emerging problem with lack of basic knowledge in physical geography, caused by reducing the time available for practice exercises, and a change in the timetable that separated lecture material from practical classes. So started experimenting with various forms of “edutainment” in evening sessions, or during fieldtrips and weekend seminars. These include quizes, a memory game, The students liked the games, and they clearly improved both motivation and knowledge.

9:30: Michael Mayer, talking about Using competence-based and project-related approaches to support students individually. He’s especially concerned about competencies such as critical thinking, work organization, teamworking, and so on. Aha! His pedagogical approach is based on constructivism. The students have to develop personal portfolios, and get regular email feedback on their individual learning style and selection of topics (plus regular grading to provide extrinsic motivation). Of course, it’s very time-consuming for the teacher. The seminars are based on development of mindmaps. 

9:45: Varyl Thorndycraft, on use of Google Earth and virtual fieldwork in teaching physical geography. There’s plenty of evidence on the value of fieldwork in teaching physical geography, but limitations (cost, safety, time, etc) on how much is possible. Varyl suggests that virtual fieldwork can supplement (but not replace) real fieldwork. Google Earth provides the ability to measure distance and elevation, to visualize spatial scales by zooming, and visualize landscapes via the tilt. But some of the height measurements are inaccurate, and there’s no control over when images were taken (and they may change from time to time, which can affect planning for virtual fieldwork). He demonstrated some uses of google earth to cover a wide range of topics, where real fieldwork would not be possible.

After some coffee (and a complaint from the person sitting next to me during that session that my keyboard is too loud!), we have this:

10:30: Thomas Stocker, from the University of Bern, and co-chair of the IPCC working group I, receiving the Hans Oeschger Medal. His talk picks up on the idea of the earth being “twitchy” that I mentioned yesterday. It’s entitled From Abrupt Change to the Future. Hmm – big audience for this one. Good job I got here early, and that my battery is fully charged, as it’s that room with no power outlets again. Okay, here we go. First a little retrospective on the work on Hans Oeschger, from his early work on ice cores to early models of CO2 exchange in the atmosphere. In 1978, long before Kyoto, Hans was publishing analysis of stabilization of CO2, including that CO2 emissions needed to peak shortly after the year 2000, and then drop dramatically, in order to stabilize concentrations in the atmosphere. The essential knowledge was available back in 1978 about what the challenge was and what we needed to do, and (looking at the Keeling curve), there’s been no progress since then on addressing the challenge. In 1984, from greenland ice cores, they demonstrated that there was a consistent signal of abrupt climate changes from different locations.

In 1984, Broeker developed a famous visualization of the ocean conveyer belt. The key idea is that the North Atlantic cools the southern hemisphere. Through the 1990’s there was a debate because the paleoclimactic record did not support the theory that the Northern and Southern hemispheres were linked in this way. Then in 2003, after reanalysis of the data, the connection between Greenland and Antarctic warming became clear, and a new very simple model of these processes was developed. This led to a much better understanding of abrupt climate change. Lessons from this story: be bold (but not too often), be persistent, and be open and critical – sometimes have to re-think the models, sometimes have to re-analyze the data to make sense of (apparently) conflicting evidence.

1987, Broeker (again) published a paper in Nature on “unpleasant surprises in the greenhouse“. Led in the 1990s to an analysis of thresholds in the more complex 3D climate models. But even in 2007, the models are all over the place in analyzing these thresholds. So how do we know what kinds of abrupt changes are in store, and whether these are irreversible? For example, plenty of people think we have already passed a threshold for arctic sea ice melting. Many regions of the world are likely to experience irreversible drought (from Solomon et al 2009).

So, to finish, some open questions: where is the geographical and physical origin of abrupt climate change in the past? What are the triggers? What are the rates of change? what are the impacts of these changes on water cycle, etc.? Are there thresholds in the climate system, and what finding in this space are robust? (if we’re too bold here it will hurt our credibility). What controls potential instabilities in the ice sheets?

11:55: Change of sessions again, to the one on Climate Prediction: Models, Diagnostics, and Uncertainty Analysis. I missed this morning’s session on model diagnostics, but just arrived in time to catch Hermann Held talking about the Effects of climate uncertainties on welfare optimal investment streams into mitigation technologies. The problem: society has to invest in mitigation strategies, but under uncertain conditions. This makes the economics assessment of welfare optimization complicated. Economists tend to ask what it would cost to constrain global temperature rise to 2°C, and discount the future costs, under the assumption that mitigation strategies impact economic growth by constraining energy use. Uncertainty over climate sensitivity has a big impact on economically optimal emissions path, which means you can generate almost any answer to the economics question by choosing a different figure for sensitivity. The study design is interesting – here’s the paper,

13:30: After lunch, the topic of the session switches to uncertainty in projections and impacts. First up: Ronald Prinn, from MIT, talking on Probabilistic Forecast for 21st Century Climate Based on Uncertainties in Emissions. Based on an interesting model (IGSM) that couples climate simulation with economics concerns, and performs uncertainty analysis on the economics aspects, especially the growth of emissions under different climate policies. Lots of interesting probability analyses – too much for me to summarize, but here’s the paper.  Oh, and he showed the ‘”wheel of fortune” I mentioned in my last post.

13:45: Andrei Sokolov, also from MIT, talking on Sensitivity of climate change projections to uncertainties in the estimates of observed changes in deep-ocean heat content. Too much detail in this talk for me to follow…

14:00: Ana Lopez, from the London School of Economics. From climate model ensembles to climate change impacts. The environment agency in the south west of England is interested in exploring the added value of ensembles of climate models in their everyday decision making, especially around water resource management. Used a perturbed physics ensemble from climateprediction.net (which is based on HadCM3), and a 21 model ensemble from CMIP3. Need to do some downscaling to get useful results to input to an impact model to make decisions about adaptation. (here’s an early paper from this work)

14:20: Next up is Stefan Fronzek’s talk on Probabilistic projections of climate change effects on sub-arctic palsa mires, in which I learnt that palsa mires are permanently frozen peat hummocks, which are sensitive to climate change, and if they melt are a major source of methane. Again, using perturbed physics ensembles from the Hadley Centre, the idea is to get probability density functions from the models to use for impact assessment. Sample result: risk of total loss of palsa mires estimated to be 80% by 2100 under scenario A1B.

14:35: Roberto Ferrise, talking on Climate Change and Projected Impacts in Agriculture: an Example on Mediterranean Crops. Starts by setting risk thresholds for minimum crop yield. Then takes climate change probability functions (again, using the Hadley Centre’s joint probabilities), and uses these as input to a crop growth model.  Here, you have to take the effect of increase CO2 (which stimulates crop growth), decreased precipitation (drought!) and increased temperature (both of which inhibit growth). The impact is very different on different crop types, and different regions across southern Europe, and is also different over time, depending on which of the various factors dominate. For example for durum wheat, we see increased yield for the next few decades, followed by loss of almost all productivity by mid century. For grapevines, the risk of lower productivity drops everywhere except in northern Italy and Northern Greece, where the risk continually rises. One of the questioners asked about uncertainty in the crop models themselves, as opposed to uncertainty in the climate. This is an interesting question, because the analysis of uncertainty in climate models looks like it is a far more mature field than analysis of uncertainty in other types of model.

14:50: Last talk in this session is by Nathan Urban on Probabilistic hindcasts and projections of the coupled climate, carbon cycle, and Atlantic meridional overturning circulation systems. Uses some simple box models (of MOC, Climate, and Carbon cycle) coupled together, to assess the probability of experiencing an MOC collapse within the next 200 years vs. triggering a collapse (i.e. setting up conditions such that a collapse will occur by 2300). The good news is that the probability of experiencing a collapse within the next 100 years remains less than 10% (consistent with the IPCC), but rises rapidly after 2100. However the probability of triggering (committing to) a collapse is about 25% in 2100 and 75% by 2200. Of course the results depend on emissions scenario used, and this is only a relatively simple climate model. (Here’s the paper).

Quick chat to the MIT folks in the coffee break to understand their development processes, especially how they cope with such a multidisciplinary team contributing to the models. Key to success (just like at Hadley) appears to be to get everyone in the same building. They do have one additional challenge which is a collaboration with the folks at Woods Hole, but as that’s only 60 miles, the Woods Hole folks make it up to weekly coordination meetings at MIT.

15:30: Okay, downstairs, to the session on Information and services technologies for Earth and Space Sciences. First speaker: Bryan Lawrence, of BADC, talking about the Moles project (Metadata Objects for Linking the Environmental Sciences). He started with a quick history of the NERC Data Grid (which includes lots of cute acronyms: COWS, MILK, MOLES). Moles: the core goal is to support users thoughout the process, where they start by trying to find data, browse if they don’t know what they’re looking for, and then manipulate the data they find in different ways. Characterize the different types of metadata involved. (quote: “we would never try to integrate discipline-specific metadata”). Teleporting & orienteering – jump to something that’s close to the kind of data you want, and then move around in the local space to find it. Define the metadata that’s common across disciplines (he walked us through some example schemas). Using a GML application schema is clearly not a sufficient condition for interoperability, and only maybe is a necessary condition, but plenty of evidence that it helps. (lots more, but he talks faster than I can type…phew.)

15:45: John Laxton, talking about GeoSciML v2: an interchange and mark-up language for geologic information. GeoSciML is a GML application schema for geology information. As Bryan said, GMLs don’t give you everything: to genuinely exchange information you need to look at the semantics. V2.0 was released at the AGU fall meeting in 2008. He’s walking through some example schemas, e.g. for Geological Feature; and Geological Unit, Geologic Structure. (and here’s a paper). Oh, and I forgot that this is supporting the OneGeology project I mentioned the other day.

16:00: The next few papers were also on various Markup Languages:  Marc Löwner, talked about GeoGML, including the challenges in developing the UML models for geological morphology, capturing both structural and process relationships between concepts. Ilya Zaslavsky talked about the CUAHSI Water Markup Language (WaterML); and Todd King talked about the SPASE Data Model.

Interdisciplinary Graduate Studies

15. April 2009 · Write a comment · Categories: collaborative science, courses, philosophy, systems thinking · Tags: , , , ,

Having talked with some of our graduate students about how to get a more inter-disciplinary education while they are in grad school, I’ve been collecting links to collaborative grad programs at U of T:

The Dynamics of Global Change Doctoral Program, housed in the Munk Centre. The core course, DGC1000H is very interesting – it starts with Malcolm Gladwell’s Tipping Point book, and then tours through money, religion, pandemics, climate change, the internet and ICTs, and development. What a wonderful journey.

The Centre for the Environment runs a Collaborative Graduate Program (MSc and PhD) in which students take some environmental science courses in addition to satisfying the degree requirements of their home department. The core course for this program is ENV1001, Environmental Decision Making, and it also include an internship to get hands on experience with environmental problem solving.

The Knowledge Media Design Institute (KMDI) also has a collaborative doctoral program, perfect for those interested in design and evaluation of new knowledge media, with a strong focus on knowledge creation, social change, and community

Finally, the Centre for Global Change Science has a set of graduate student awards, to help fund grad students interested in global change science. Oh, and they have a fascinating seminar series, mainly focussed on climate science (all done for this year, but get on their mailing list for next years seminars).

Are there any more I missed?

Agent-based simulations

14. April 2009 · Write a comment · Categories: climate modeling, climate science, systems thinking · Tags: , , , , , , , , ,

Had an interesting conversation this afternoon with Brad Bass. Brad is a prof in the Centre for Environment at U of T, and was one of the pioneers of the use of models to explore adaptations to climate change. His agent based simulations explore how systems react to environmental change, e.g. exploring population balance among animals, insects, the growth of vector-borne diseases, and even entire cities. One of his models is Cobweb, an open-source platform for agent-based simulations. 

He’s also involved in the Canadian Climate Change Scenarios Network, which takes outputs from the major climate simulation models around the world, and extracts information on the regional effects on Canada, particularly relevant for scientists who want to know about variability and extremes on a regional scale.

We also talked a lot about educating kids, and kicked around some ideas for how you could give kids simplified simulation models to play with (along the line that Jon was exploring as a possible project), to get them doing hands on experimentation with the effects of climate change. We might get one of our summer students to explore this idea, and Brad has promised to come talk to them in May once they start with us.

Oh, and Brad is also an expert on green roofs, and will be demonstrating them to grade 5 kids at the Kids World of Energy Festival.

Reinventing Computer Science

13. April 2009 · 1 comment · Categories: collaborative science, philosophy, systems thinking · Tags: , , , ,

Computer Science, as an undergraduate degree, is in trouble. Enrollments have dropped steadily throughout this decade: for example at U of T, our enrollment is about half what it was at the peak. The same is true across the whole of North America. There is some encouraging news: enrollments picked up a little this year (after a serious recruitment drive, ours is up about 20% from it’s nadir, while across the US it’s up 6.2%). But it’s way to early to assume they will climb back up to where they were. Oh, and percentage of women students in CS now averages 12% – the lowest ever.

What happened? One explanation is career expectations. In the 80’s, its was common wisdom that a career in computers was an excellent move, for anyone showing an aptitude for maths. In the 90’s, with the birth of the web, computer science even became cool for a while, and enrollments grew dramatically, with a steady improvement in gender balance too. Then came the dotcom boom and bust, and suddenly a computer science degree was no longer a sure bet. I’m told by our high school liaison team that parents of high school students haven’t got the message that the computer industry is short of graduates to recruit (although with the current recession that’s changing again anyway).

A more likely explanation is perceived relevance. In the 80’s, with the birth of the PC, and in the 90’s with the growth of the web, computer science seemed like the heart of an exciting revolution. But now computers are ubiquitous, they’re no longer particularly interesting. Kids take them for granted, and a only a few über-geeks are truly interested in what’s inside the box. But computer science departments continue to draw boundaries around computer science and its subfields in a way that just encourages the fragmentation of knowledge that is so endemic of modern universities.

Which is why an experiment at Georgia Tech is particularly interesting. The College of Computing at Georgia Tech has managed to buck the enrollment trend, with enrollment numbers holding steady throughout this decade. The explanation appears to be a radical re-design of their undergraduate degree, into a set of eight threads. For a detailed explanation, there’s a white paper, but the basic aim is to get students to take more ownership of their degree programs (as opposed to waiting to be spoonfed), and to re-describe computer science in terms that make sense to the rest of the world (computer scientists often forget the the field is impenetrable to the outsider). The eight threads are: Modeling and simulation; Devices (embedded in the physical world); Theory; Information internetworks; Intelligence; Media (use of computers for more creative expression); People (human-centred design); and Platforms (computer architectures, etc). Students pick any two threads, and the program is designed so that any combination covers most of what you would expect to see in a traditional CS degree.

At first sight, it seems this is just a re-labeling effort, with the traditional subfields of CS (e.g. OS, networks, DB, HCI, AI, etc) mapping on to individual threads. But actually, it’s far more interesting than that. The threads are designed to re-contextualize knowledge. Instead of students picking from a buffet of CS courses, each thread is designed so that students see how the knowledge and skills they are developing can be applied in interesting ways. Most importantly, the threads cross many traditional disciplinary boundaries, weaving a diverse set of courses into a coherent theme, showing the students how their developing CS skills combine in intellectually stimulating ways, and preparing them for the connected thinking needed for inter-disciplinary problem solving.

For example the People thread brings in psychology and sociology, examining the role of computers in the human activity systems that give them purpose. It explore the perceptual and cognitive abilities of people as well as design practices for practical socio-technical systems. The Modeling and Simluation thread explores how computational tools are used in a wide variety of sciences to help understand the world. Following this thread will require consideration of epistemology of scientific knowledge, as well as mastery of the technical machinery by which we create models and simulations, and the underlying mathematics. The thread includes in a big dose of both continuous and discrete math, data mining, and high performance computing. Just imagine what graduates of these two threads would be able to do for our research on SE and the climate crisis! The other thing I hope it will do is to help students to know their own strengths and passions, and be able to communicate effectively with others.

The good news is that our department decided this week to explore our own version of threads. Our aims is to learn from the experience at Georgia Tech and avoid some of the problems they have experienced (for example, by allowing every possible combination of 8 threads, it appears they have created too many constraints on timetabling and provisioning individual courses). I’ll blog this initiative as it unfolds.