How to apply Continuous Integration practices with Business Intelligence and Analytics practices

1. 1
Continuous Integration
For
Business Intelligence
And Analytics
Alex Meadows
Principal Consultant
CSpring

2. 2
Agenda
● About Me
● Waterfall vs Agile
● Continuous Integration
● Software Release Process
● References
Feel free to ask questions at any point in the presentation!

3. 3
Alex Meadows
● Worked in various
industries (manufacturing,
social/email marketing,
consulting, software,
agriculture)
● Open source agilest
● Loves data :)

4. 4
Let's Talk Releases...
●What's your team's release candance?
●How many bug releases go out afterwards?
●How much new development gets cut?

5. 5
Waterfall Approach
Source: http://covalentmarketing.com/blog/2013/02/25/software-development-lifecycle-
waterfall-vs-agile/

6. 6
Agile Approach
Source: http://www.mountaingoatsoftware.com/system/asset/file/17/ScrumLargeLabelled.png

7. 7
Continuous Integration
● Release Early, Release Often!
● Bundle work together into cohesive “builds”
– Server Configuration
– Code
– Database
– Tests, tests, and more tests
– Approval Process

8. 8
Continuous Integration
● Agile XP principal since 1994
● Used in software everywhere*
– Google
● Tests 100 million test cases every day
● Entire code base changes 50% monthly
– Amazon, Yahoo, Microsoft, etc.
* Used in mature agile software shops

9. 9
Software Release Process
Source: http://www.360logica.com/blog/2014/06/continuous-integration.html

10. 10
Plan
● Break tasks out into manageable chunks
– Can that ETL code be broken down by:
● Source/Target
● Table
● Process
– Can that dashboard be broken down by:
● Object
● Type of data/visual/interaction
● Business must prioritize, team must commit

11. 11
BI Challenges
● Work is difficult to break apart
– Most tools are designed for waterfall
– Complex moving components

12. 12
Code
● Team works and develops “artifacts”
– “Code” - Reports, ETL, Visualizations, Database
changes, etc.
– Documentation
– Research

13. 13
Build
● Developed work is
– Run through unit tests
– Peer reviewed
– “Compiled” - tested for readiness for QA
– Pushed to QA

14. 14
BI Challenges
● Coordination across disparate tools, components
● Version control doesn't coordinate between tools
● Database version control is hard to introduce

15. 15
Test
● Verify all functionality, edge cases, and user requirements
● 10% manual, 90% automation work
● Automated gate keeper (pass/fail the build)

16. 16
BI Challenges
● Test tools not written for many components
– Database – exists, but minimalist
– ETL – database tools and etlTest (shameless plug ;)
– Visualization – Selenium
– Reports – mix of the above
– Analytics – mix of the above

17. 17
Release/Deploy
● Coordinate and push to next environment
● Approvals act as gate keeping mechanisim
● Cycle starts back and the code phase as needed

18. 18
BI Challenges
● Coordination is possible, but challenging
● Tools might not provide integration points

19. 19
Operate/Monitor
● Release into Production
● Monitor processes, logs, etc for aberrations
● Bugs and other issues are added into backlog for future
releases

20. 20
References

21. 21
Sources
● http://www.eclipsecon.org/2013/sites/eclipsecon.org.2013/files/
●