Will RSpec help us to do better Science?                                             Maximilian Ott                       ...
© 2011 NICTA   2
Why are we doing what we are doing?     © 2011 NICTA                     3
Curious Bunch - Want to figure out new things      © 2011 NICTA                              4
The advancement of science dependsin general on the interplay betweenexperimental studies and theory                      ...
Many interesting Things only happen at Scale      © 2011 NICTA                                          Young Hyun, CAIDA ...
Need for LARGE               Experimental Facilities© 2011 NICTA                             7
© 2011 NICTA   8
© 2011 NICTA   9
Hypothesis               Analysis                 Experiment                          Observation© 2011 NICTA             ...
Hypothesis               Analysis                 Experiment                          Observation© 2011 NICTA             ...
How “well” do we do experiments today?                                                    58%                             ...
How can we improve this?•  The Art & Science of Experimentation   –  The art of designing experiments   –  Scientifically ...
Need for Methodologies & Best Practices                                  Observations                      Model 1        ...
Need to teach Experimenting at Scale                                           Experiment                                 ...
Need new ways to disseminate Findings                     ?   Authors are invited to submit   Full Papers of up to 10 page...
LabWiki: Publishing Your Lab Book               Portal                                                                    ...
Notes                              Experiment                                                  Description                ...
Notes                              Experiment                                                  Description                ...
Notes                              Experiment                                                  Description                ...
Summary  Researcher                 Comment                                  SummaryReader                     Review     ...
We are working on it•  OMF   –  OEDL to describe experiments   –  Experiment Controller to run them•  OML   –  Capture mea...
Portal                                 OMFWiki                                                                            ...
Call to Action•  The Art & Science of Experimentation   –  The art of designing experiments   –  Scientifically sound meth...
omf.mytestbed.net               wiki.mytestbed.net               max.ott@nicta.com.au© 2011 NICTA                         ...
Upcoming SlideShare
Loading in...5
×

Will RSpec help us to do better science

170

Published on

These are the slides for a plenary talk I gave at GEC 11 in Denver in July 2011. The primary message is that we, the networking community need to fundamentally improve the way we do experimental research.

Published in: Technology, Education
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
170
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
1
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Will RSpec help us to do better science

  1. 1. Will RSpec help us to do better Science? Maximilian Ott With lots of help from Thierry Rakotoarivelo & Guillaume Jourjon© 2011 NICTA. All Rights Reserved. from imagination to impact
  2. 2. © 2011 NICTA 2
  3. 3. Why are we doing what we are doing? © 2011 NICTA 3
  4. 4. Curious Bunch - Want to figure out new things © 2011 NICTA 4
  5. 5. The advancement of science dependsin general on the interplay betweenexperimental studies and theory Wikipedia © 2011 NICTA 5
  6. 6. Many interesting Things only happen at Scale © 2011 NICTA Young Hyun, CAIDA 6
  7. 7. Need for LARGE Experimental Facilities© 2011 NICTA 7
  8. 8. © 2011 NICTA 8
  9. 9. © 2011 NICTA 9
  10. 10. Hypothesis Analysis Experiment Observation© 2011 NICTA 10
  11. 11. Hypothesis Analysis Experiment Observation© 2011 NICTA 11
  12. 12. How “well” do we do experiments today? 58% (19) 30% (10) 21% 12% 9% (7) (4) (3) Missing Missing Missing Missing OK Model Experiment Replication Precision Validation Description Information Information “Evaluation” section of SIGCOMM 2010 papers © 2011 NICTA 12
  13. 13. How can we improve this?•  The Art & Science of Experimentation –  The art of designing experiments –  Scientifically sound methodology & analysis•  Dissemination –  Overcoming the 6 page paper information barrier•  Learning –  Reflection on the methodology –  Developing Best Practices•  Tools –  To streamline processes –  To record processes and outcomes © 2011 NICTA 13
  14. 14. Need for Methodologies & Best Practices Observations Model 1 … Model 2 Ha , H0 Ha , H0 Experiment Experiment New Obs + Analysis © 2011 NICTA 14
  15. 15. Need to teach Experimenting at Scale Experiment Queue High Speed Student Dedicated Network Network Internet In the Box Control Remote Network Experiment Server Tested application and protocol Measurement points Student ireel.npc.nicta.com.au © 2011 NICTA 15
  16. 16. Need new ways to disseminate Findings ? Authors are invited to submit Full Papers of up to 10 pages and Testbed Practices Papers of up to 6 pages, including references, figures and tables, formatted according to the IEEE conference format. © 2011 NICTA 16
  17. 17. LabWiki: Publishing Your Lab Book Portal Studied Phenomenon Model 1 ... Model i HA , H 0 HA , H 0 Experiment Experiment OEDL Experiment Experiment Script Experiment Experiment Measure- ments Observations database 200 350000 Distance Bandwidth 180 300000 160 140 250000 R Distance (m) 120 Analyses Bandwidth 200000 100 150000 80 Script 60 40 100000 50000 20 0 0 Falsify and/or 0 50 100 150 200 250 300 Time (s) refine the models © 2011 NICTA 17
  18. 18. Notes Experiment Description Observations Model 1 … Model 2 Code Ha , H0 Ha , H0 Runs Experiment Experiment New Obs + Analysis Analysis Measurements© 2011 NICTA 18
  19. 19. Notes Experiment Description Observations Model 1 … Model 2 Code Ha , H0 Ha , H0 Runs Experiment Experiment New Obs + Analysis Analysis Measurements© 2011 NICTA 19
  20. 20. Notes Experiment Description Observations Model 1 … Model 2 Code Runs Ha , H0 Summary Ha , H0 Experiment Experiment New Obs + Analysis Analysis Measurements© 2011 NICTA 20
  21. 21. Summary Researcher Comment SummaryReader Review Analysis Reviewer © 2011 NICTA 21
  22. 22. We are working on it•  OMF –  OEDL to describe experiments –  Experiment Controller to run them•  OML –  Capture measurements & meta-data•  Portal –  Wiki: Keep notes with references –  Schedule & monitor runs –  Analysis with integrated R server –  Repository for all artifacts © 2011 NICTA 22
  23. 23. Portal OMFWiki OML ON/OFF e1 App 1 Measurement Library (OML) Proxy e2 Server Control App 2 Network OML SQL c1 ... Server Database ... Sensing daemon sensor device System Monitor Experiment Node Control Node OEDL Visualization R Link Quality: Very Good Good Poor Very Poor Y/Z Coordintate X Coordinate© 2011 NICTA 23
  24. 24. Call to Action•  The Art & Science of Experimentation –  The art of designing experiments –  Scientifically sound methodology & analysis•  Dissemination –  There need to be better ways than 10 page papers•  Learning –  Reflection on the methodology –  Developing Best Practices –  Teaching experimentation at scale•  Tools –  To streamline processes –  To record processes and outcomes © 2011 NICTA 24
  25. 25. omf.mytestbed.net wiki.mytestbed.net max.ott@nicta.com.au© 2011 NICTA 25
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×