Serendipity

Friday, the last day of the main conference, kicked off with Pamela Zave’s keynote “Software Engineering for the Next Internet”. Unfortunately I missed the first few minutes of the talk. But I regret that, because this was an excellent keynote. Why do I say that? Because Pamela demonstrated a beautiful example of what I want to call “software systems thinking”. By analyzing them from a software engineering perspective, she demonstrated how some of the basic protocols of the internet (eg the Simple Initiation Protocol, SIP), and the standardization process by which they are developed are broken in interesting ways. The reason they are broken is because they ignore software engineering principles. I thought the analysis was compelling: both thorough in terms of the level of detail, and elegant in the simplicity of the analysis.

Here’s some interesting tidbits;

• A corner case is a possible behaviour that emerges from the interaction of unanticipated constraints. It is undesirable, and designers typically declare it to be rare and unimportant, without any evidence. Understanding and dealing with corner cases is important for assessing the robustness of a design.
• The IETF standards process is an extreme (pathological?) case of bottom up thinking. It sets an artificial conflict between generality and simplicity, because any new needs are dealt with by adding more features and more documents to the standard. Generality is always achieved by making the design more complex. Better abstractions, and some more top down analysis can provide simple and general designs (and Pamela demonstrated a few)
• How did the protocols get to be this broken? Most network functions are provided by cramming them into the IP layer. This is believed to be more efficient, and in the IETF design process, efficiency always takes precedence over separation of concerns.
• We need a successor to the end-to-end principle. Each application should run on a stack of overlays that exactly meets its requirements. Overlays have to be composable. The bottom underlay runs on a virtual network which gets a predictable slice of the real network resources. Of course, there are still some tough technical challenges in designing the overlay hierarchy.

So, my reflections. Why did I like this talk so much? First it had an appealing balance of serious detail (with clear explanations) and new ideas that are based on an understanding of the big picture. Probably it helps that she’s talking about an analysis approach using techniques that I’m very familiar with (some basic software engineering design principles: modularity, separation of concerns, etc), and applies them to a problem that I’m really not familiar with at all (detailed protocol design). So that combination allows me to follow most of the talk (because I understand the way she approaches the problem), but tells me a lot of things that are new and interesting (because the domain is new to me).

She ended with a strong plug for domain-specific research. It’s more fun and more interesting! I agree wholeheartedly with that. Much of software engineering research is ultimately disappointing because in trying to be too generic it ends up being vague and wishy washy. And it misses good pithy examples.

So, having been very disappointed with Steve McConnell’s opening keynote yesterday, I’m pleased to report that the keynotes got steadily better over the week. Thursday’s keynote was by Carlo Ghezzi, entitled Reflections on 40+ years of software engineering research and beyond: An Insider’s View. He started with a little bit of history of the SE research community and the ICSE conference, but the main part of the talk was a trawl though the data from the conference over the years, motivated by questions such as “how international are we as a community?”, and “how diverse?” (e.g. academia, industry…), and “how did the research areas included in ICSE evolve?”. For example, there has been a clear trend in the composition of the program committee, from being N. American dominated (80% at first ICSE), to now approx equal N. American and European, with some from asia & elsewhere. However, there is a startling trend on industry vs. acadmia mix. The attendees at the first conference were 80% industry and only 20% academics. This has steadily changed: the conference is now 90% academics. The number of accepted papers each year has remained fairly steady (average is 44), but with a strong growth in submissions over past 15 years from 150 to 400. Which now gives us a paper acceptance rate now well below 15%. This is clearly good for the academics – the low acceptance rate keeps the quality of the accepted papers high, and makes the conference the top choice as a publication venue. But a strong academic research program clearly does not attract practitioners to attend.

In Carlo’s analysis of research areas, I was struck by the graph of number of papers on programming languages, which looks like a pair of vampire teeth – a huge spike in this area in the early days of ICSE, then nothing for years, and again a huge spike in the last couple of years. A truly interesting and surprising result.

Towards the end of the talk, Carlo got onto the question of how we could identify our best products. He talked about the strengths and weaknesses of quantitative measures such as citation count (difficult as it’s a moving target, and you have to account for journal/conference versions), number of downloads from ACM digital library over 12 months, etc. He drew a lot on a report by the Joint Committee on Quantitative Assessment of Research. He also mentioned Meyer’s viewpoint article in CACM April 2009, and of course, Parnas’s somewhat less nuanced “Stop the numbers game“. Why is the problem of quantitative assessment of research becoming so hot today? It’s being increasingly used to rank journals and conferences and individuals. Many stakeholders now need to evaluate research, and peer-review is considered to be expensive and subjective, while numeric metrics are considered to be simple and objective. The Joint committee report says that, to the contrary, numeric metrics are simple and misleading. From the report: Much of modern bibliometrics is flawed. The meaning of a citation can be even more subjective than peer review. Citation counts are only valid if reinforced by other judgements.

Carlos’ final message was that we have to care about impact of our research: understanding, measuring, and improving it. Because if we don’t others will (governments, funding agencies, universities, etc). Okay, that’s a good argument. I’ve been skeptical of SIGSOFT’s Impact Project in the past, largely because I think the process by which research ideas filter into industrial practice is much more complex, and takes much longer than everyone seems to expect. But I guess taking control of the assessment of impact is the obvious way to address this issue.

After the break, Jorge presented his paper on the Secret Life of Bugs. His did a great job on presenting the work, to an absolutely packed room, and I had lots of people comment on how much they enjoyed the paper afterwards. I beamed with pride.

But for most of the day, I was busy trying to finish off my talk “Software Engineering for the Planet”, in time for the session at 2pm. Many thanks to Spencer, Jon, Carolyn and Alicia for helping my polish it prior to delivery. I’ll get the slides up on the web soon. I think the session went very well – the questions and discussions afterwards were very encouraging – most people seemed to immediately get the key message (that we should stop focussing our energies on personal green choices, and instead figure out how our professional skills and experience can be used to address to the climate crisis). Aran posted a quick summary of the session, and some afterthoughts. Now we’ve got to do the community building, and keep the momentum going. [Aran said he doesn’t think I’ll get much research done in the next few months. He’s might be right, but I can just declare that this is now my research…]

Okay, the main conference started today, and we kick off with the keynote speaker – Steve McConnell talking about “The 10 most powerful ideas in software engineering”. Here’s my thoughts: when magazines are short of ideas for an upcoming issue, they resort to the cheap journalist’s trick of inventing top ten lists. It makes for easy reading filler, that never really engages the brain. Unfortunately, this approach also leads to dull talks. The best keynotes have a narrative thread. They tell a story. They build up ideas in interesting new ways. The top ten format kills this kind of narrative stone dead (except perhaps when used in parody). Okay, so I didn’t like the format, but what about the content? Steve walked through ten basic concepts that we’ve been teaching in our introductory software engineering courses for years, so I learned nothing new. Maybe this would be okay as a talk to junior programmers who missed out on software engineering courses in school. For ICSE keynotes, I expect a lot more – I’d have liked at least some sharper insights, or better marshalling of the evidence. I’m afraid I have to add this to my long list of poor ICSE keynotes. Which is okay, because ICSE keynotes always suck – even when the chosen speakers are normally brilliant thinkers and presenters. Maybe I’ll be proved wrong later this week… For what it’s worth, here’s his top ten list (which he said were in no particular order):

1. Software Development work is performed by human beings. Human factors make a huge difference in the performance of a project.
2. Incrementalism is essential. The benefits are feedback, feedback, and feedback! (on the software, on the development process, on the developer capability). And making small mistakes that prevent bigger mistakes later.
3. I’ve no idea what number 3 was. Please excuse my inattention.
4. Cost to fix a defect increases over time, because of the need to fix all the collateral and downstream consequences of the error.
5. There’s an important kernel of truth in the waterfall model. Essentially, there are three intellectual phases: discovery, invention, construction. They are sequential, but also overlapping.
6. Software Estimates can be improved over time, by reducing its uncertainty as the project progresses.
7. The most powerful form of reuse is full reuse – i.e. not just code and design, but all aspects of process.
8. Risk management is important.
9. Different kinds of software call for different kinds of software development (the toolbox approach). This was witty: he showed a series of pictures of different kinds of saw, then newsflash: software development is as difficult as sawing.
10. The software engineering body of knowledge (SWEBOK)

Next up, the first New Ideas and Emerging Results session. This is a new track at this year’s ICSE, and the intent is to have a series of short talks, with a poster session at the end of the day. Although I’m surprised how hard it was to get a paper accepted: of 118 submissions, they selected only 21 for presentation (an 18% acceptance rate). The organisers also encouraged the presenters to use the Pecha Kucha format: 20 slides on an automated timer, with 20 seconds per slide. Just to make it more fun and more dynamic.

I’m disappointed to report that none of the speakers this morning took up this challenge, although Andrew Begel’s talk on social networking for programmers was very interesting (and similar to some of our ideas for summer projects this year). The fourth talk, by Abram Hindle, also didn’t use the Pecha Kucha format, but made up for it with a brilliant and beautiful set of slides that explain how to form interesting time series analysis visualizations of software projects by mining the change logs.

Buried in the middle of the session was an object lesson in misuse of empirical methods. I won’t name the guilty parties, but let me describe the flaw in their study design. Two teams were assigned a problem to analyze, with one team being given a systems architecture, and the other team wasn’t. To measure the effect of being given this architecture on the requirements analysis, the authors asked experts to rate each of several hundred requirements generated by each of the teams, and then used a statistical test to see whether the requirements from one team were different on this ranking compared to the other. Unsurprisingly, they discovered a statistically significant difference. Unfortunately, the analysis is completely invalid, because they made a classic unit of analysis error. The unit of analysis for the experimental design is the team, because it was teams that were assigned the different treatments. But the statistical test was applied to individual requirements. But there was no randomization of these requirements – all the requirements from a given team have to be taken as a single unit. The analysis that was performed in this study merely shows that the requirements came from two different teams, which we knew already. It shows nothing at all about the experimental hypothesis. I guess the peer review process has to let a few klunkers through.

Well, we reach the end of the session and nobody did the Pecha Kucha thing. Never mind – my talk is first up in the next NIER session this afternoon, and I will take the challenge. Should be hilarious. On the plus side, I was impressed with the quality of all the talks – they all managed to pack in key ideas, make them interesting, and stick to the 6 minute time slot.

So, I made it to ICSE at last. I’m way behind on blogging this one: the students from our group have been here for several days, busy blogging their experiences. So far, the internet connection is way too weak for liveblogging, so I’l have to make do with post-hoc summaries.

I spent the morning at the Socio-Technical Congruence (STC) workshop. The workshops is set up with discussants giving prepared responses to each full paper presentation, and I love the format. The discussants basically riff on ideas that the original paper made them think of. Which ends up being more interesting than the original paper. For example, Peri Tarr clarified how to tell whether something counts as a design pattern. A design pattern is a (1) proven solution to a (2) commonly occurring problem in a (3) particular context. To assess whether an observed “pattern” is actually a design pattern, you need to probe whether all these three things are in place. For example, the patterns that Marcelo had identified do express implemented solutions, but e has not yet identified the problems/concerns they solve, and the contexts in which the patterns are applicable.

Andy Begel’s discussion include a tour through learning theory (I’ve no idea why, but I enjoyed the ride!). On a single slide, he tooks us though the traditional “empty container” model of learning, though Piaget‘s constructivism; Vygotsky‘s social learning, Papert‘s constructionism), Van Maanen & Schein‘s newcomer socialization; Hutchins‘ distributed cognition and Lave & Wenger‘s legitimate peripheral participation. Whew. Luckily, I’m familiar with all of these except the Van Maanen & Schein stuff – I’m looking forward to read that. Oh and an interesting book recommendation “Anything that’s worth knowing is really complex” from Wolfram’s A New Kind of Science. Then, Andy posed some interesting question’s: how long can software live? How big can it get? How many people can work on it? And he proposed we should design for long-term social structures, rather than modular architecture.

We then spent some time discussing whether designing the software architecture is the same thing as designing the social structure. Audris suggested that while software architecture people tend not to talk about the social dimension, but in fact they are secretly designing it. If the two get out of synch, people are very adaptable – they find a way of working around the mismatch. Peri pointed out that technology also adapts to people. They are different things, with feedback loops that affect each other. It’s an emergent, adaptive thing.

And someone mentioned Rob DeLine’s keynote on the weekend at CHASE, in which pointed out that only about 20% of ICSE papers mention the human dimension, and we should seek to flip the ratio. To make it 80% we should insist that papers that ignore the people aspects have to prove that people are irrelevant to the problem being addressed. Nice!

After lots of catching up with ICSE regulars over lunch, I headed over to the last session of the Michael Jackson festschrift, to hear Michael’s talk. He kicked off with some quotes that he admitted he can’t take credit for: “description should precede invention”, and Tony Hoare’s: “there are 2 ways to make a system (1) make it so complicated that it has no obvious deficiencies or (2) make it so simple that it obviously has no deficiencies”. And another which may or may not be original: “Understanding is a process, not a state”. And another interesting book recommendation: Personal Knowledge by Michael Polanyi.

So, here’s the core of MJ’s talk: every “contrivance” has an operational principle, which specifies how the characteristic parts fulfill their function. Further, knowledge of physics, chemistry, etc, is not sufficient to understand and recognise the operating principle. E.g. describing a clock – the description of the mechanism is not a scientific description. While the physical science has made great strides, our description of contrivances has not. The operational principle answers questions like “What is it?” “What is it for?”,  and “how do the parts interact to achieve the purpose?”. To supplement this, the mathematical and scientific knowledge describes the underlying laws, context necessary for success (e.g. pendulum clock only works in the appropriate gravitational field, and must be completely upright – won’t work on the moon, on a ship, etc), part properties necessary for success, possible improvements, specific failures and causes, feasibility of a proposed contrivance.

MJ then goes on to show how problem decomposition works:

(1) Problem decomposition –  by breaking out the problem frames: e.g. for an elevator: provide prioritized lift service, brake on danger, provide information display for users.

(2) Instrumental decomposition – building manager specifies priority rules, system uses priority rules to determine operation.

The sources of complexity are the intrinsic complexity of each subproblem, plus the interaction of subproblems. But he calls for the use of free decomposition (meaning free as in unconstrained). For initial description purposes, there are no constraints on how the subproblems will interact; the only driver is that we’re looking for simple operating principles.

Finally, then he identified some composition concerns: interleaving (edit priority rules vs lift service); requirements elaboration (e.g. book loans vs member status), requirements conflict (linter-library vs member loan), switching (lift service vs emergency action), domain sharing (e.g. phone display: camera vs gps vs email).

The discussion was fascinating, but I was too busy participating to take notes. Hope someone else did!