A very informal talk I gave to Hausi Muller's group at UVic in June 2009. I have included, without permission, slides from Daniel Hook's excellent presentation at SE-CSE 2009 (http://www.cs.ua.edu/~SECSE09/schedule.htm).

1. Why do climate modellers trust their software? Jon Pipitone Advisor: Steve Easterbrook University of Toronto @Uvic, June 2009

4. What am I missing?

5. "If we knew what we were doing, it wouldn't be called research, would it?" - attributed to Albert Einstein

6. What is climate modelling? A kind of computational science

9. Program outputs are the results of the experiment.

10. Scientific software development my focus

12. What is climate modelling? Source: Easterbrook, CUSEC'09 (Source: IPCC AR4, 2007)

13. What is climate modelling? Source: Easterbrook, CUSEC'09 (Source: IPCC AR4, 2007)

14. General Circulation Models Source: Easterbrook, CUSEC'09 © Crown Copyright

15. Scientific software development

16. Scientific software development

19. Model-intercomparisons Source: Easterbrook, CUSEC'09

21. Validation Notes Source: Easterbrook, CUSEC'09 © Crown Copyright

22. Core problems with V&V

23. Validation notes Bit reproducibility Core problems with V&V

25. What does this mean for software quality?

26. (note, we're not talking about model quality!)

27. How do we judge quality in scientific software?

33. Model crashes during a run

34. Model runs, but variables drift out of tolerance

36. The right results for the “wrong reasons” (e.g. over-tuning)

37. Expected improvement not achieved Source: Easterbrook, CUSEC'09

38. Measuring Quality: Defect Density So, Is climate modelling software really that good?

40. When are we counting: pre- or post-release?

42. No standards in the literature

46. What makes a piece of software bad ?

47. How do you know when you're done?

48. How do you train newcomers?

49. or...

50. When have you had to delay releasing due to a bug? Why?

51. Tell me the story behind these and other bugs...

54. Already have connections with CM groups

58. What do climate modellers do when coding to guarantee correctness?

59. What are their notions of quality wrt to code?

60. How can we benchmark computational scientists' code quality?

63. Breakout by defect types (but, what are they?)

64. Maybe: static fault density using automated tool

65. Examine several climate models (>3?)

67. Use questions given previously to guide conversation

68. Investigate the story of a defect, judgement calls that were made.

69. ~5 defect stories per climate modelling centre

70. Cross-case analysis

74. … objective the study?