The yt Project:Growing & Engaging a Community of Practice                                     Matthew Turk                ...
What is yt?
astro-ph/1011.3514astro-ph/1112.4482    yt-project.org
yt is a platform for analysis andvisualization of astrophysical* simulations
install script:    Full dependency stack             Source code Development environment                     GUI          ...
There are many simulation codes.
There are many simulation codes,         but there’s only one sky.
data structures,       methods,   assumptions,     IO formats,           units,variable names,               ...
Fully-Supported   Semi-Supported   In-Progress     Enzo            Chombo         Cactus    FLASH             Athena      ...
yt is designed to address physical,                not computational,             entities and questions.
Orthogonal Rays                      1DNon-orthogonal RaysSlicesOblique Slices        2DProjectionsSpheresRectangular Pris...
Scripting interface, HTML5 GUI,in-situ processing, transparentmulti-level parallelism...
yt is supposed to get out of the way.
from yt.mods import *pf = load(’galaxy0030/galaxy0030’)p = SlicePlot(pf, 2, ’Density’, ’c’, (200,’kpc’))p.save(’Galaxy’)
10   -12           g/cc
6     C.B. Hummels et al.Figure 1. Sample projections from the medium feedback and cooling suppression run at z = 0.5. Eac...
Canned Analysis Tasks
Absorption SpectrumCoordinate Transformations      Halo Finding     Mass Functions      Merger Trees      Halo Profiling   ...
“Community”?
Traditional View    “Users” “Developers”
Most Scientific Software          “Users”“Developers”
Community of Practice     “Devusers”
“Developers”
“Developers”Inspection and verification
“Developers”Inspection and verificationTracking modifications
“Developers”Inspection and verificationTracking modificationsSharing information
“Developers”Inspection and verificationTracking modificationsSharing informationAdding functionality
“Developers”Inspection and verificationTracking modificationsSharing informationAdding functionalityAll are necessary charac...
“Users”
“Users”Uncritical acceptance of code...?
“Users”Uncritical acceptance of code...?“These are people we give the code  to that don’t care how it works.”
Developing as a Team
Challenges
Reward Structure
de facto and de jure
de facto and de jure•Utilization of developed tools•Respect from community•Project involvements•Invitations to speak
de facto and de jure•Funding•Publications•Citation count•Influence
Traditional astrophysics does   not favor tool builders.
Chores
ChoresDocumentation,         testing,      outreach, infrastructure.
Tasks not fully-aligned with reward   structure goals present great      motivational challenges.
Co-opetition
Co-opetition(                      )    •Funding    •Publications    •Citation count    •Influence
How developer community engagement -- as  seen in cohesion, excitement, energy -- isaffected by funded improvements is unc...
Strategies
The Art of Community by BaconProducing Open Source Software by FogelTeam Geek by Fitzpatrick & Collins-SussmanOrganizing S...
You must design the community you want.
Technical & Social
Technical & Social
SVN                             hg             Users &              Repos                   Users &                       ...
Forky development:very low barrier to entry;everything comes in the box.
Testing:answer as well as integration testsget run every 30 minutes.
Code review:eyes on (nearly) every changesetmentoring new developers
The path upstream must be obvious.
•Happy application•Itch-scratching•Submission•Code review & mentoring•Participation
Non-core contributed code:a place for sharing scripts,data and images
Communication
Communication   All business is conducted openly.
Communication            Immediate
2   Communication               Low-Latency
Communication           High-Latency
Technical & Social
HRT
HumilityRT
HumilityRespectT
HumilityRespectTrust
HumilityRespectTrust           (Fitzpatrick & Collins-Sussman)
The most important thing I try toremember is that I want to foster a community of peers, not of elites.
“ I’ve noticed something is acting  strangely with ...                 “
“ I’ve noticed something is acting  strangely with ...                 ““ You’re probably doing it wrong.                 ...
“ I’ve noticed something is acting  strangely with ...                 ““ Can you tell us how you would  expect it to act?...
“ I think that there might be a bug in  the way ...                            “
“ I think that there might be a bug in  the way ...                             ““ It’s like that for very good reasons.  ...
“ I think that there might be a bug in  the way ...                            ““ It behaves the way it does  because ... ...
Thoughtfulness,       inquiry,   confidence,    letting go.
This culture must be seeded directly.
This culture must be seeded directly,because culture will self-propagate.
letting go
Pride rather than ownership
Projects passing between people
Too much control can smother.
willToo much control can smother.
Successes
Developed by working astrophysicists.
Tom Abel            Jean-Claude PassyDavid Collins       Thomas RobitailleBrian Crosby        Anna RosenAndrew Cunningham ...
Tom Abel            Jean-Claude PassyDavid Collins       Thomas RobitailleBrian Crosby        Anna RosenAndrew Cunningham ...
Usage on NSF                                                NICS Nautlius     “Usage of a HPC Data Analysis andVisualizati...
(play movie)
“... it seems likely that significant softwarecontributions to existing scientific softwareprojects are not likely to be rew...
Thank you.
Tom Abel            Jean-Claude PassyDavid Collins       Thomas RobitailleBrian Crosby        Anna RosenAndrew Cunningham ...
yt: Growing and Engaging a Community of Practice
yt: Growing and Engaging a Community of Practice
yt: Growing and Engaging a Community of Practice
yt: Growing and Engaging a Community of Practice
yt: Growing and Engaging a Community of Practice
yt: Growing and Engaging a Community of Practice
yt: Growing and Engaging a Community of Practice
yt: Growing and Engaging a Community of Practice
yt: Growing and Engaging a Community of Practice
yt: Growing and Engaging a Community of Practice
yt: Growing and Engaging a Community of Practice
yt: Growing and Engaging a Community of Practice
yt: Growing and Engaging a Community of Practice
yt: Growing and Engaging a Community of Practice
yt: Growing and Engaging a Community of Practice
yt: Growing and Engaging a Community of Practice
Upcoming SlideShare
Loading in...5
×

yt: Growing and Engaging a Community of Practice

154
-1

Published on

The yt project ( http://yt-project.org/ ) is a community developed analysis and visualization system for astrophysical simulation data. In this presentation I talk a bit about what yt is, and then discuss the challenges and strategies for growing a community of practice.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
154
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
6
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

yt: Growing and Engaging a Community of Practice

  1. 1. The yt Project:Growing & Engaging a Community of Practice Matthew Turk Columbia University
  2. 2. What is yt?
  3. 3. astro-ph/1011.3514astro-ph/1112.4482 yt-project.org
  4. 4. yt is a platform for analysis andvisualization of astrophysical* simulations
  5. 5. install script: Full dependency stack Source code Development environment GUI Sample data
  6. 6. There are many simulation codes.
  7. 7. There are many simulation codes, but there’s only one sky.
  8. 8. data structures, methods, assumptions, IO formats, units,variable names, ...
  9. 9. Fully-Supported Semi-Supported In-Progress Enzo Chombo Cactus FLASH Athena Hydra Orion ART Gadget Nyx RAMSES Uniform Data GDF Piernik
  10. 10. yt is designed to address physical, not computational, entities and questions.
  11. 11. Orthogonal Rays 1DNon-orthogonal RaysSlicesOblique Slices 2DProjectionsSpheresRectangular PrismsDisks/CylindersInclined Boxes 3DClumpsExtracted RegionsEllipsoids
  12. 12. Scripting interface, HTML5 GUI,in-situ processing, transparentmulti-level parallelism...
  13. 13. yt is supposed to get out of the way.
  14. 14. from yt.mods import *pf = load(’galaxy0030/galaxy0030’)p = SlicePlot(pf, 2, ’Density’, ’c’, (200,’kpc’))p.save(’Galaxy’)
  15. 15. 10 -12 g/cc
  16. 16. 6 C.B. Hummels et al.Figure 1. Sample projections from the medium feedback and cooling suppression run at z = 0.5. Each image represent a column-density
  17. 17. Canned Analysis Tasks
  18. 18. Absorption SpectrumCoordinate Transformations Halo Finding Mass Functions Merger Trees Halo Profiling Level Sets Light Cones Light Rays Time Series Star Analysis Two-Point Analysis
  19. 19. “Community”?
  20. 20. Traditional View “Users” “Developers”
  21. 21. Most Scientific Software “Users”“Developers”
  22. 22. Community of Practice “Devusers”
  23. 23. “Developers”
  24. 24. “Developers”Inspection and verification
  25. 25. “Developers”Inspection and verificationTracking modifications
  26. 26. “Developers”Inspection and verificationTracking modificationsSharing information
  27. 27. “Developers”Inspection and verificationTracking modificationsSharing informationAdding functionality
  28. 28. “Developers”Inspection and verificationTracking modificationsSharing informationAdding functionalityAll are necessary characteristics of the scientific process as a whole.
  29. 29. “Users”
  30. 30. “Users”Uncritical acceptance of code...?
  31. 31. “Users”Uncritical acceptance of code...?“These are people we give the code to that don’t care how it works.”
  32. 32. Developing as a Team
  33. 33. Challenges
  34. 34. Reward Structure
  35. 35. de facto and de jure
  36. 36. de facto and de jure•Utilization of developed tools•Respect from community•Project involvements•Invitations to speak
  37. 37. de facto and de jure•Funding•Publications•Citation count•Influence
  38. 38. Traditional astrophysics does not favor tool builders.
  39. 39. Chores
  40. 40. ChoresDocumentation, testing, outreach, infrastructure.
  41. 41. Tasks not fully-aligned with reward structure goals present great motivational challenges.
  42. 42. Co-opetition
  43. 43. Co-opetition( ) •Funding •Publications •Citation count •Influence
  44. 44. How developer community engagement -- as seen in cohesion, excitement, energy -- isaffected by funded improvements is unclear.
  45. 45. Strategies
  46. 46. The Art of Community by BaconProducing Open Source Software by FogelTeam Geek by Fitzpatrick & Collins-SussmanOrganizing Simulation Code Collectives by SundbergScientific Software Production by Howison & Herbsleb
  47. 47. You must design the community you want.
  48. 48. Technical & Social
  49. 49. Technical & Social
  50. 50. SVN hg Users & Repos Users & ReposRepository Users & Repos Users & Repos Users & Repos Users & Users Repos
  51. 51. Forky development:very low barrier to entry;everything comes in the box.
  52. 52. Testing:answer as well as integration testsget run every 30 minutes.
  53. 53. Code review:eyes on (nearly) every changesetmentoring new developers
  54. 54. The path upstream must be obvious.
  55. 55. •Happy application•Itch-scratching•Submission•Code review & mentoring•Participation
  56. 56. Non-core contributed code:a place for sharing scripts,data and images
  57. 57. Communication
  58. 58. Communication All business is conducted openly.
  59. 59. Communication Immediate
  60. 60. 2 Communication Low-Latency
  61. 61. Communication High-Latency
  62. 62. Technical & Social
  63. 63. HRT
  64. 64. HumilityRT
  65. 65. HumilityRespectT
  66. 66. HumilityRespectTrust
  67. 67. HumilityRespectTrust (Fitzpatrick & Collins-Sussman)
  68. 68. The most important thing I try toremember is that I want to foster a community of peers, not of elites.
  69. 69. “ I’ve noticed something is acting strangely with ... “
  70. 70. “ I’ve noticed something is acting strangely with ... ““ You’re probably doing it wrong. “
  71. 71. “ I’ve noticed something is acting strangely with ... ““ Can you tell us how you would expect it to act? “
  72. 72. “ I think that there might be a bug in the way ... “
  73. 73. “ I think that there might be a bug in the way ... ““ It’s like that for very good reasons. Don’t touch it. “
  74. 74. “ I think that there might be a bug in the way ... ““ It behaves the way it does because ... “
  75. 75. Thoughtfulness, inquiry, confidence, letting go.
  76. 76. This culture must be seeded directly.
  77. 77. This culture must be seeded directly,because culture will self-propagate.
  78. 78. letting go
  79. 79. Pride rather than ownership
  80. 80. Projects passing between people
  81. 81. Too much control can smother.
  82. 82. willToo much control can smother.
  83. 83. Successes
  84. 84. Developed by working astrophysicists.
  85. 85. Tom Abel Jean-Claude PassyDavid Collins Thomas RobitailleBrian Crosby Anna RosenAndrew Cunningham Anthony ScopatzNathan Goldbaum Devin SilviaCameron Hummels Sam SkillmanJi-hoon Kim Stephen SkorySteffen Klemer Britton SmithKacper Kowalik Geoffrey SoMichael Kuhlen Casey StarkEve Lee Elizabeth TaskerChris Malone Matthew TurkChris Moody Rick WagnerAndrew Myers John WiseJeff Oishi John ZuHone
  86. 86. Tom Abel Jean-Claude PassyDavid Collins Thomas RobitailleBrian Crosby Anna RosenAndrew Cunningham Anthony ScopatzNathan Goldbaum Devin SilviaCameron Hummels Sam SkillmanJi-hoon Kim Stephen SkorySteffen Klemer Britton SmithKacper Kowalik Geoffrey SoMichael Kuhlen Casey StarkEve Lee Elizabeth TaskerChris Malone Matthew TurkChris Moody Rick WagnerAndrew Myers John WiseJeff Oishi John ZuHone
  87. 87. Usage on NSF NICS Nautlius “Usage of a HPC Data Analysis andVisualization System,” Szczepanski et al 2012
  88. 88. (play movie)
  89. 89. “... it seems likely that significant softwarecontributions to existing scientific softwareprojects are not likely to be rewarded throughthe traditional reputation economy of science.Together these factors provide a reason toexpect the over-production of independentscientific software packages, and the under-production of collaborative projects in whichlater academics build on the work of earlierones.” Howison & Herbsleb (2011)
  90. 90. Thank you.
  91. 91. Tom Abel Jean-Claude PassyDavid Collins Thomas RobitailleBrian Crosby Anna RosenAndrew Cunningham Anthony ScopatzNathan Goldbaum Devin SilviaCameron Hummels Sam SkillmanJi-hoon Kim Stephen SkorySteffen Klemer Britton SmithKacper Kowalik Geoffrey SoMichael Kuhlen Casey StarkEve Lee Elizabeth TaskerChris Malone Matthew TurkChris Moody Rick WagnerAndrew Myers John WiseJeff Oishi John ZuHone
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×