Investing in a good software factory and automating the build process
•Download as PPTX, PDF•
2 likes•1,039 views
Talk from a Singapore Spring user group. Investing in a software factory, automating it as much as possible. Can we also automate the creation of the software factory?
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Investing in a good software factory and automating the build process
Similar to Investing in a good software factory and automating the build process (20)
More from Nicolas Mas
More from Nicolas Mas (8)
Recently uploaded
Recently uploaded (20)
Investing in a good software factory and automating the build process
- 1. How important it is to invest in a good software factory and automate the build process as much as possible ...if possible Nicolas Mas - Nicolas.Mas@gmail.com https://www.linkedin.com/in/nicolasmas
- 2. Acknowledgment “Alone we can do so little; together we can do so much” Eric Domingo Shuang Xia Lokesh Boghte Cedric Arnoult Andrew Tiu
- 3. What this talk won’t do ⬗ Elaborate on Yet Another Software Factory Definition ⬘ Or only if, as a consultant once told me: “a common language needs to be defined to understand each other” © Ajit Alwe ;-) ⬗ Enumerate a list of tools ⬗ Talk about any PM methodology ⬗ Show any code in action ⬗ Entertain you with funny pictures (sorry)
- 4. So what is it going to be? ⬗ A discussion about a real experience ⬗ back to basic: breaking down a complex problem into smaller chunks and finding solution ⬗ An example of software factory designed to handle a specific problem ⬗ Bonus: Discuss if it’s worth automating the build of the software factory as well as the automation of the build process (inception style) ⬗ B Tracks: if we really have time
- 5. Actually we have two topics here... ...and they are complementary 1. Investing in a good software factory 2. Automate the build process as much as possible
- 6. Investing in a good software factory: YES YOU CAN! Part One:
- 7. Quick Survey ⬗ What’s a software factory for you? ⬗ Are you using one?
- 8. For us, it all started with a unique context... ⬗ A Java Spring project ⬘ Team of 5 developers ◆ A “A Team”, with distinct and partially overlapping skills and experience ⬘ A delivery manager ◆ NOT from a technical background ⬘ A product owner on the client side ◆ NOT from a technical background ◆ NOT collocated ⬘ Tight deadline
- 9. ...which induced ⬗ Small team with high pressure ⬗ Delivery manager does not speak the same language as the developers ⬗ Product owner in another country ⬗ Customer dev. processes are very different ⬗ Deliverables: ⬘ Code quality ⬘ Documentation ⬘ processes on top of the usuals ⬗ Start from scratch ⬗ A bit of innovation
- 10. So we needed to: Rethink the way we would build the product & Customize the process to build the product WHY?
- 11. To manage the risks generated by these factors & Make sure we deliver an amazing great product which will provide our customer the greatest satisfaction (hint: bullshit)
- 12. Moving on from there We can segregate our problem by vertical domains/aspects: ⬗ The product scope ⬗ Delivery management ⬗ Development process ⬘ Dependency Management ⬘ Code repository ⬘ Build process ⬘ Software Quality ⬘ Deployment process ⬗ Delivery process ⬗ Feedback ⬗ Communication
- 13. Mapping ⬗ The product scope ⬗ Delivery management ⬗ Development process ⬘ Dependency Management ⬘ Code repository ⬘ Build process ⬘ Software Quality ⬘ Deployment process ⬗ Delivery process ⬗ Feedback ⬗ Communication => Jira => Human (yes!) => Maven => GitHub => Jenkins => Sonar => Jenkins => Shell => Jira => Slack
- 14. A few remarks ⬗ The tools chosen are NOT the only one available: ⬘ Actually many !! ⬗ Rather than being a fixed structure, the Software Factory is a living system and must be design accordingly ⬗ Setup, test before the development starts ⬗ Complete transparency within the team and with the client
- 15. What did we gain doing this? ⬗ We are confident we can deliver anything, anytime ⬗ Newcomers to the project are not a potential problem ⬗ We minimized the impact of human factor ⬗ The delivery pipe is unlikely to be a blocking factor for the project ⬘ We won’t be able to put it on the tools if we fail
- 16. Is there anything else we can improve? ⬗ We designed our software factory in such a way that our main point of failure is the human factor How could we minimize that?
- 17. Automate as much as possible the build process: YES YOU CAN! (BUT...) Part Two:
- 18. What we can’t automate ⬗ code commit by the developers ⬗ stakeholders communication ⬘ at least partially (Slack technically allows us to do it, but can’t adapt to the stakeholder feedback).
- 19. What we decided not automate ⬗ Code review ⬗ Delivery ⬘ (YES this can be discussed … and YES I am aware of continuous delivery).
- 20. So finally, what we can automate The build process + UAT deployment “Jenkins is an open source continuous integration tool written in Java. Jenkins provides continuous integration services for software development. It is a server-based system running in a servlet container such as Apache Tomcat.”
- 21. ⬗ A maven build is triggered (code pulled from github) ⬗ Shell scripts deploy on UAT targets ⬗ Postprocess action: email + Slack message
- 22. So finally, what we can automate The software quality - SonarQube “SonarQube is an open platform for continuous inspection of code quality”
- 23. Code coverage
- 24. Code Quality
- 25. So how did it go? ⬗ Anytime delivery was real ⬘ BUT complexity increased with time and integration with other systems (what a surprise!) ⬗ Code coverage tools sides effects ⬘ Require dev to code in a sometimes not so efficient manner ⬗ Client communication is better ⬘ Scope management ⬘ improved transparency Delivery happened no matter what
- 26. Was it worth it? YES
- 27. Bonus: Automating the creation and setup of the software factory Can or Cannot? Part Three:
- 28. Automating the whole thing ⬗ We want to be able to recreate from a master the whole software factory stack ⬘ In case of a crash (but can it be stateful?) ⬘ A new project is started in //, with similar constraints ⬗ We want to maintain a backup infra just in case ⬘ How do we keep them aligned? ⬘ AWS + local VBOx for example
- 29. Our current setup (manual) What we manually setup so far DB Stack CI Stack Staging Stack Github virtualbox vagrant Physical server Ubuntu server Ubuntu serverUbuntu server
- 30. Dev machine: MB Pro ⬗ Eclipse IDE + JDK 1.7 ⬗ Spring 4.0.4 + Apache CXF ⬗ Maven 3.0.3 ⬗ Tomcat 7 ⬗ Findbugs Staging Server: VM: Ubuntu 14.04 LTS ⬗ JDK 1.7.0_11 ⬗ Tomcat 7.0.37 DB Server: VM: Ubuntu 14.04 LTS ⬗ Oracle Entreprise Edition 11.2.0.3 ⬗ Gemfire CI Server: VM: Ubuntu 14.04 LTS ⬗ JDK 1.7.0_11 ⬗ Jenkins ⬗ Archiva ⬗ GitHub ⬗ Jira Agile ⬗ Jira Crucible ⬗ Sonar Cube 4.1.2 Infra: Ubuntu 14.04 LTS ⬗ Vagrant ⬗ Saltstack ⬗ packer ⬗ Virtualbox
- 31. Our target through automation DB Stack CI Stack Staging Stack Github virtualbox vagrant AWS Physical server WebApp - Software Factory Builder Ubuntu serverUbuntu serverUbuntu server
- 32. The Software Factory Builder ⬗ Node.js (locally hosted) front end ⬗ Underlying technologies (main ones) ⬘ Packer (OS baking) ⬘ SaltStack (provisioning) ⬘ Vagrant (VM management)
- 33. What it looks like ⬗ A node.js interface ⬘ You can imagine anything for the VM type (VMWare, digital ocean etc.)
- 34. What it looks like
- 35. What it looks like
- 36. What it looks like
- 37. Pros & Cons What’s cool.. ⬗ Traditionally, software factories are built once and used for projects for years. Stacks are updated, patched, reconfigured => lower and lower reliability. In our case, we can rebuild it every time ⬗ We can duplicate on different infrastructures local or in the cloud ⬗ Once defined, the build pattern is fast to execute What’s not… ⬗ Maintaining the setup and testing it before a “gold” version takes time and is complexe => it’s an upfront investment you might not benefit from ⬗ You can’t automate 100%, rather 90%. So you still need a bit of work
- 38. Bonus Track - Did you know ? “The mother of all demos” http://en.wikipedia.org/wiki/The_Mother_of_All_Demos ⬗ Douglas Engelbart's December 9, 1968 ⬗ 90-minute live public demonstration of the online system, NLS, they had been working on since 1962 ⬗ 1000 attendees and a demo of almost all features we use today ⬘ Mouse ⬘ hypertext, object addressing and dynamic file linking ⬘ shared-screen collaboration involving two persons at different sites communicating over a network with audio and video interface. http://web.stanford.edu/dept/SUL/library/extra4/sloan/mousesite/1968Dem o.html
- 40. Bye Bye ! Nicolas Mas - Nicolas.Mas@gmail.com https://www.linkedin.com/in/nicolasmas
Editor's Notes
- How important it is to invest in a good software factory and automate the build process as much as possible. This is what is going to keep us busy for the next 45 minutes.
- First, this is a team effort. So here is the team.
- Why? list of tools: Because it does not make sense, and we would only stay at the surface of the topic PM Methodo: we want this discussion being still relevant no matter which past/present/future PM approach is used Code in action: too complex and murphy’s law !Funny pics: Because @Lokesh said so, and like with Chuck Norris, you obey
- A discussion: something that actually happened Back to basics: we are software developers, we have one very usual way to solve problems Example of SF: What we did in a project, how we came to this specific set of tools
- History: since grad, only notepad, ne IDE and the JVM. It did not prevent me to graduate in computer science. projects at that time: one person project, no customer, all poc only constraint was the delivery. Which means the delivery itself does not justify a soft factory. When does this statement deprecates? Is there one version of the software factory?
- So we had this java spring project to do. A team 5 developers, an agile delivery manager (who is not from a technical background), one product owner on the client side who is not collocated and a very tight.
- Underlying factors: small team with high pressure (what happens if one is sick?) Delivery manager does not speak the same language as the developers Product owner in another country Customer development processes are different Deliverables includes code quality, documentation, processes among the usual Start from scratch A bit of innovation
- All theses asked us to rethink the way we would build the product and customize the process to build the product.
- Why? - or let’s put it simply: we want to survive the project… Delivering is the drive here. We need to make sure we deal we what we can manage in order to be able to handle all the unexpected things which will pour on the project.. + murphy’s law
- Let’s do it the developper way: let’s break the big problem into smaller chunk and solve them: we can segregate by vertical domains (the obvious one) This is important because it will allow us to map the proper tool with the right problem to solve
- This becomes our software factory
- Choice of tools: a few examples (github/SVN/Perforce,Gradel,TeamCity, ECL Emma … etc.) Our SF is actually a living system which is evolving with the product being developed: all the component can be changed/removed/adapated. It should be non intrusive.
- Back to our product: the SF made us confident to deliver anything, anytime. We minimized the impact of human factor: which is happening only when required. The delivery pipe is unlikely to be a blocking factor
- Now our main point of failure is the human factor. We can address as much as possible by automating everything we can within the software factory.
- Build process in a general manner of speaking
- [LB] We can automate stakeholder communication a bit with Slack, with feeds from JIRA and Jenkins and Github, some part of stakeholder communication (at least the technical part) can be automated.
- Then there are some things that we decided NOT to be automated: Code review
- examples Build process (jenkins) (+ examples) Software Quality (ECL EMMA) (+ examples) Deployment process (shell script) (+ examples)
- Many tools: (squale, metricsware, kalistix, cast etc.)
- This is a screenshot from Sonar Code Coverage plugin, it works perfectly well with standard JUnit which means that its easy to add on, we do not have to make any change to the way we write unit tests. Sonar uses ECL EMMA for Code Coverage analysis. Sonar also has IDE plugins (Ecliipse, IntelliJ, etc) so doing this sort of analysis within IDE during development is one click. On the top you can see that this class has a 62.5% coverage, what that means is out of all the lines of codes written in that class, the unit tests we have execute about 63% lines and generally speaking are safe because they are tested. It does not check whether unit tests are of any meaning or not. One can just write some unit tests and not have any assertions and still have this number high, which really does not mean anything :) It can highlight code that is not covered and code that is covered. It can also show branch and loop coverage, which means that it will show all possible conditional coverage. Which is a good thing again if unit tests are meaningful. M
- This is Sonar with Findbugs plugin. It has a lot of good coding rules and if some code violates that it will highlight them like throwing java.lang.Exception for example. It also categorises them as Critical, Major etc. Sonar itself has other add on features to manage the issues. This has plus and minus, few things identified here are good but in long term this start to somehow force developer to write code in a different way which does not feel good.
- So how did it go? (how this SF helped us to address our concerns) Anytime delivery was real. However It tends to degrade the anytime delivery Code coverage tools: example: to increase the test coverage, we have tests which do not assert anything. Client communication is better: We do know what is going on.
- Was the effort to think, design and build the software factory worth it? made us explore tech we use but never try to understand reusability scalability SF architecture: virtualized & automated Getting it out of the way
- So far the creation of the SF is manual, and time consuming. Can we build the software factory in an automated manner?
- [TODO] short description of the infra this time.
- Describe what the WebApp does. Dynamically setup and configure the virtual stack
- We first thought of automating the build after all the conf files are generated, but decided to manually call the tools to build the stack (it’s a quick prototype).
- Bottom line: it’s an interesting concept but more at the R&D level Middle ground: automate what is easy to automate, plus what is always used. Alternative: CoreOS + binaries in Docker containers ?