Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Extreme Programming for ETL and Data Analytics

1,656 views

Published on

Over the last decade, eXtreme Programming practices like User Stories, Evolutionary Design, Test-Driven Development (TDD), Behavior Driven Developer (BDD), Refactoring, Continuous Integration and Automation have fundamentally changed software development processes and inherently how engineers work.

Having experienced various benefits from XP practices on our J2EE stack, our team started to apply these practices to extract, transform, and load (ETL) and Data Analytics side of our product. Unfortunately, there is very little guidance available in this context, esp. for the SAS Platform. Right from finding the unit testing framework to structuring the code to designing our modules and setting up a Continuous Integration builds, our team had to figure out everything, the hard way.

Join us to understand the challenges we faced during this process and how we resolved these challenges.

Details: http://confengine.com/agile-pune-2014/proposal/535/extreme-programming-for-etl-and-data-analytics

Conference: http://pune.agileindia.org/

Published in: Software
  • Softcrylic’s digital analytics service focuses on implementing data tracking techniques and real-time engagement techniques to drive traffic, improve conversions and maximize ROI. http://bit.ly/2Daudsb
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Extreme Programming for ETL and Data Analytics

  1. 1. eXtremeProgramming for ETL Agile Pune 2014 _________________________ Prafulla Girgaonkar [@pgirgaonkar] Naresh Jain [@nashjain] Illustration: Umesh Kale
  2. 2. 25YEARS 6000+CLIENTS Nr 1M ROOMS 94COUNTRIES 95%+ ago we started helping clients manage their revenue later we’re still finding the revenue opportunity in every situation around the world are priced every single day using IDeaS on 6 continents have hotels that now run on IDeaS Revenue Management Systems More than 95 percent of all our clients are still working with us today. About IDeaS
  3. 3. ETL DECISION IP PMS/CRS Competitor WebRate Social Media Periodically or Real Time Full or Delta Messages or File * * * * ETL & IDEAS
  4. 4. Concept Exploration Structuring Requirements Discovery Earlier ETL Practices
  5. 5. Concept DataExploration “Structurization” ETLDevelopment Certification GA Typical Development Cycle on an ETL Project
  6. 6. Vendor IDEAS Development Earlier ETL Practices
  7. 7. Challenges and Flaws Source : www.dogri.org/
  8. 8. Challenges and Flaws
  9. 9. Challenges and Flaws
  10. 10. Challenges and Flaws
  11. 11. Challenges and Flaws
  12. 12. Challenges and Flaws Time Discovery Development
  13. 13. Challenges and Flaws Source : http://roselawgroupreporter.com/
  14. 14. Challenges and Flaws
  15. 15. Requirement Conformity Requirement Misunderstanding Late Data Change Delays API Performance Priorities Challenges and Flaws -Vendor
  16. 16. Naresh Jain •nareshjain.com •@nashjain
  17. 17. PrafullaS Girgaonkar Product Development for Hospitality |Revenue Optimization |Finance Prafulla.Girgaonkar@ideas.com @pgirgaonkar
  18. 18. Journey Hilstar Opera Pegasus
  19. 19. Old Approach
  20. 20. Existing Approach -Iterative & Incremental Approach
  21. 21. Initial XP Practices -Hilstar
  22. 22. DBAQADEVPMUX Initial XP Practices –Whole Team Source: http://www.selfishprogramming.com/ BOKM
  23. 23. Iteration #1 Iteration #2 Iteration #3 Initial XP Practices –Small Releases
  24. 24. Initial XP Practices -Continuous Integration
  25. 25. Initial XP Practices –User Story
  26. 26. Evolution of the Practices… OPERA Source: http://www.freedomchurchunited.org/
  27. 27. Evolution of the Practices -Pairing
  28. 28. Evolution of the Practices –Planning/Grooming Source: http://www.h4-entertainment.com/
  29. 29. Evolution of the Practices –Test Driven Development
  30. 30. Evolution of the Practices –Refactoring
  31. 31. Evolution of the Practices –Coding Standards
  32. 32. Evolution of the Practices –Collective Ownership
  33. 33. Evolution of the Practices –Simple Design
  34. 34. •The customer is always available •Code theunit testfirst •Leaveoptimizationfor later Evolution of the Practices –Optimization at the End
  35. 35. Write a TEST before FIXING the BUG Evolution of the Practices –Test Driven Bug Fixing
  36. 36. Further…Evolution of the Practices… Pegasus
  37. 37. Data Management 4GL MACROS PROCS FUNCTIONS Learn SAS 4gl @ http://support.sas.com/software/ products/ondemand- academics/ Further…Evolution of the Practices –SAS 4GL
  38. 38. Further…Evolution of the Practices –Earlier Code
  39. 39. No Unit test No Build Integration Language Unawareness Discover the Tool SASUNIT Further…Evolution of the Practices –Testability
  40. 40. Further…Evolution of the Practices –Code Later
  41. 41. Part of build Process Fail Fast Further…Evolution of the Practices –Build Process Correction
  42. 42. Integrated With Jenkins Fail Fast Fail Reports Further…Evolution of the Practices –Jenkins for SASUNIT
  43. 43. Engineering Practices are always at the core. Tools May not be there. There is WILL, there is WAY CRUX
  44. 44. Benefits Source: http://flickr.com
  45. 45. Source: http://docustar.com/blog/quality-check-is-99-9-good-enough/ Benefits –
  46. 46. Source: http://www.ftjco.com/wp-content/uploads/2012/10/diamonds.jpg Benefits
  47. 47. Benefits
  48. 48. Benefits
  49. 49. Benefits Relative Time Hilstar Opera Pegasus
  50. 50. UNIT Test … UNIT Test … UNIT Test … Source: www.topnews.in Benefits
  51. 51. Source: http://en.wikipedia.org/wiki/Cello Benefits
  52. 52. Change Management Source: http://www.animalnational.com/monarch-butterfly-migration/
  53. 53. Change Management –Usual Reactions Source: http://ericgerlach.com/introphilosophy8/ DEVs | QAs | POs | BOs
  54. 54. Change Management –Looking for Rituals Source: http://www.theguardian.com/books/2014/apr/27/reading-print-books-ereaders-technology DEVs | QAs | POs | BOs
  55. 55. Courtesy: Jeff Patton Change Management –Smaller Release/Stories were Shaker DEVs | QAs | POs | BOs
  56. 56. Change Management –Incremental Development DEVs | QAs | POs | BOs
  57. 57. Change Management –Story Map DEVs | QAs | POs | BOs
  58. 58. Change Management –Spikes were new Source: http://www.kitchology.com/ DEVs | QAs
  59. 59. Change Management –CI & Failing Builds DEVs | QAs
  60. 60. Change Management –Pairing Source: https://www.airpair.com/mean-stack/posts/2014-10-job-post-mean-stack-developer DEVs | QAs
  61. 61. Change Management –Team Ownership DEVs | QAs Source: https://dilbert.com
  62. 62. Change Management –Testers part of Design Discussion Source: http://www.mercatornet.com/articles/view/health_economics_101_contraceptives_pay_for_themselves_usually_babies_dont DEVs | QAs
  63. 63. Change Management –Defect are discussed; not entered QAs Source: http://letstalkaboutdiscussion.ca/
  64. 64. Change Management –Test Automation QAs Source: http://www.mobileqazone.com/profiles/blogs/software-testing-by-human-or-machine-what-should-shouldn-t-be
  65. 65. Road Ahead Source: http://www.gettyimages.in/
  66. 66. Engineering Practice -Refactoring / Managing Code Smells Source:www.123rf.com
  67. 67. •Engineering Practice -Simple Design & Solving critical challenges by taking them out of context Road Ahead -Engineering Practice –Simple Design Source: http://flickr.com/
  68. 68. •Engineering Practice -Simple Design & Solving critical challenges by taking them out of context Road Ahead -Engineering Practice –Sandbox Source: http://ifum.com/
  69. 69. Road Ahead -Engineering Practice -Evolutionary Design Extended Source: http://nordicedge.com/
  70. 70. Road Ahead -Engineering Practice –Programming skills and polyglot
  71. 71. Road Ahead -Real Time Integration
  72. 72. WEI JAD Road Ahead -Joint Application development with Vendor Source: http://www.p-r-i.org/nadcap/nadcap-meetings/
  73. 73. Take Away •Engineering Practices are universal –determination and perseverance •XP Practices lead to T-Shaped individual or Generalizing Specialist •Product Discovery & Story Mapping vital for Alignment.
  74. 74. Prafulla Girgaonkar [@pgirgaonkar] Naresh Jain [@nashjain] Illustration: Umesh KaleSession Ends.. Efforts Continue…

×