Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

yt: Growing and Engaging a Community of Practice

263 views

Published on

The yt project ( http://yt-project.org/ ) is a community developed analysis and visualization system for astrophysical simulation data. In this presentation I talk a bit about what yt is, and then discuss the challenges and strategies for growing a community of practice.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

yt: Growing and Engaging a Community of Practice

  1. 1. The yt Project:Growing & Engaging a Community of Practice Matthew Turk Columbia University
  2. 2. What is yt?
  3. 3. astro-ph/1011.3514astro-ph/1112.4482 yt-project.org
  4. 4. yt is a platform for analysis andvisualization of astrophysical* simulations
  5. 5. install script: Full dependency stack Source code Development environment GUI Sample data
  6. 6. There are many simulation codes.
  7. 7. There are many simulation codes, but there’s only one sky.
  8. 8. data structures, methods, assumptions, IO formats, units,variable names, ...
  9. 9. Fully-Supported Semi-Supported In-Progress Enzo Chombo Cactus FLASH Athena Hydra Orion ART Gadget Nyx RAMSES Uniform Data GDF Piernik
  10. 10. yt is designed to address physical, not computational, entities and questions.
  11. 11. Orthogonal Rays 1DNon-orthogonal RaysSlicesOblique Slices 2DProjectionsSpheresRectangular PrismsDisks/CylindersInclined Boxes 3DClumpsExtracted RegionsEllipsoids
  12. 12. Scripting interface, HTML5 GUI,in-situ processing, transparentmulti-level parallelism...
  13. 13. yt is supposed to get out of the way.
  14. 14. from yt.mods import *pf = load(’galaxy0030/galaxy0030’)p = SlicePlot(pf, 2, ’Density’, ’c’, (200,’kpc’))p.save(’Galaxy’)
  15. 15. 10 -12 g/cc
  16. 16. 6 C.B. Hummels et al.Figure 1. Sample projections from the medium feedback and cooling suppression run at z = 0.5. Each image represent a column-density
  17. 17. Canned Analysis Tasks
  18. 18. Absorption SpectrumCoordinate Transformations Halo Finding Mass Functions Merger Trees Halo Profiling Level Sets Light Cones Light Rays Time Series Star Analysis Two-Point Analysis
  19. 19. “Community”?
  20. 20. Traditional View “Users” “Developers”
  21. 21. Most Scientific Software “Users”“Developers”
  22. 22. Community of Practice “Devusers”
  23. 23. “Developers”
  24. 24. “Developers”Inspection and verification
  25. 25. “Developers”Inspection and verificationTracking modifications
  26. 26. “Developers”Inspection and verificationTracking modificationsSharing information
  27. 27. “Developers”Inspection and verificationTracking modificationsSharing informationAdding functionality
  28. 28. “Developers”Inspection and verificationTracking modificationsSharing informationAdding functionalityAll are necessary characteristics of the scientific process as a whole.
  29. 29. “Users”
  30. 30. “Users”Uncritical acceptance of code...?
  31. 31. “Users”Uncritical acceptance of code...?“These are people we give the code to that don’t care how it works.”
  32. 32. Developing as a Team
  33. 33. Challenges
  34. 34. Reward Structure
  35. 35. de facto and de jure
  36. 36. de facto and de jure•Utilization of developed tools•Respect from community•Project involvements•Invitations to speak
  37. 37. de facto and de jure•Funding•Publications•Citation count•Influence
  38. 38. Traditional astrophysics does not favor tool builders.
  39. 39. Chores
  40. 40. ChoresDocumentation, testing, outreach, infrastructure.
  41. 41. Tasks not fully-aligned with reward structure goals present great motivational challenges.
  42. 42. Co-opetition
  43. 43. Co-opetition( ) •Funding •Publications •Citation count •Influence
  44. 44. How developer community engagement -- as seen in cohesion, excitement, energy -- isaffected by funded improvements is unclear.
  45. 45. Strategies
  46. 46. The Art of Community by BaconProducing Open Source Software by FogelTeam Geek by Fitzpatrick & Collins-SussmanOrganizing Simulation Code Collectives by SundbergScientific Software Production by Howison & Herbsleb
  47. 47. You must design the community you want.
  48. 48. Technical & Social
  49. 49. Technical & Social
  50. 50. SVN hg Users & Repos Users & ReposRepository Users & Repos Users & Repos Users & Repos Users & Users Repos
  51. 51. Forky development:very low barrier to entry;everything comes in the box.
  52. 52. Testing:answer as well as integration testsget run every 30 minutes.
  53. 53. Code review:eyes on (nearly) every changesetmentoring new developers
  54. 54. The path upstream must be obvious.
  55. 55. •Happy application•Itch-scratching•Submission•Code review & mentoring•Participation
  56. 56. Non-core contributed code:a place for sharing scripts,data and images
  57. 57. Communication
  58. 58. Communication All business is conducted openly.
  59. 59. Communication Immediate
  60. 60. 2 Communication Low-Latency
  61. 61. Communication High-Latency
  62. 62. Technical & Social
  63. 63. HRT
  64. 64. HumilityRT
  65. 65. HumilityRespectT
  66. 66. HumilityRespectTrust
  67. 67. HumilityRespectTrust (Fitzpatrick & Collins-Sussman)
  68. 68. The most important thing I try toremember is that I want to foster a community of peers, not of elites.
  69. 69. “ I’ve noticed something is acting strangely with ... “
  70. 70. “ I’ve noticed something is acting strangely with ... ““ You’re probably doing it wrong. “
  71. 71. “ I’ve noticed something is acting strangely with ... ““ Can you tell us how you would expect it to act? “
  72. 72. “ I think that there might be a bug in the way ... “
  73. 73. “ I think that there might be a bug in the way ... ““ It’s like that for very good reasons. Don’t touch it. “
  74. 74. “ I think that there might be a bug in the way ... ““ It behaves the way it does because ... “
  75. 75. Thoughtfulness, inquiry, confidence, letting go.
  76. 76. This culture must be seeded directly.
  77. 77. This culture must be seeded directly,because culture will self-propagate.
  78. 78. letting go
  79. 79. Pride rather than ownership
  80. 80. Projects passing between people
  81. 81. Too much control can smother.
  82. 82. willToo much control can smother.
  83. 83. Successes
  84. 84. Developed by working astrophysicists.
  85. 85. Tom Abel Jean-Claude PassyDavid Collins Thomas RobitailleBrian Crosby Anna RosenAndrew Cunningham Anthony ScopatzNathan Goldbaum Devin SilviaCameron Hummels Sam SkillmanJi-hoon Kim Stephen SkorySteffen Klemer Britton SmithKacper Kowalik Geoffrey SoMichael Kuhlen Casey StarkEve Lee Elizabeth TaskerChris Malone Matthew TurkChris Moody Rick WagnerAndrew Myers John WiseJeff Oishi John ZuHone
  86. 86. Tom Abel Jean-Claude PassyDavid Collins Thomas RobitailleBrian Crosby Anna RosenAndrew Cunningham Anthony ScopatzNathan Goldbaum Devin SilviaCameron Hummels Sam SkillmanJi-hoon Kim Stephen SkorySteffen Klemer Britton SmithKacper Kowalik Geoffrey SoMichael Kuhlen Casey StarkEve Lee Elizabeth TaskerChris Malone Matthew TurkChris Moody Rick WagnerAndrew Myers John WiseJeff Oishi John ZuHone
  87. 87. Usage on NSF NICS Nautlius “Usage of a HPC Data Analysis andVisualization System,” Szczepanski et al 2012
  88. 88. (play movie)
  89. 89. “... it seems likely that significant softwarecontributions to existing scientific softwareprojects are not likely to be rewarded throughthe traditional reputation economy of science.Together these factors provide a reason toexpect the over-production of independentscientific software packages, and the under-production of collaborative projects in whichlater academics build on the work of earlierones.” Howison & Herbsleb (2011)
  90. 90. Thank you.
  91. 91. Tom Abel Jean-Claude PassyDavid Collins Thomas RobitailleBrian Crosby Anna RosenAndrew Cunningham Anthony ScopatzNathan Goldbaum Devin SilviaCameron Hummels Sam SkillmanJi-hoon Kim Stephen SkorySteffen Klemer Britton SmithKacper Kowalik Geoffrey SoMichael Kuhlen Casey StarkEve Lee Elizabeth TaskerChris Malone Matthew TurkChris Moody Rick WagnerAndrew Myers John WiseJeff Oishi John ZuHone

×