Download to read offline
More Related Content
Similar to The_Little_Jenkinsfile_That_Could
The_Little_Jenkinsfile_That_Could
- 1. The Little Jenkinsfilethat Could @ShelleyMLambert June 2019 Themes from the story: Impossible task Hard work Optimism “I think I can, I think I can”
- 2. Outline Background Historyof Proprietary Tools & Technical Debt Scope, Stats, Ecosystem Design Principles for Automation Intentionally Simple Learn from Past Lessons Whole team involvement Revision Highlights of the little Jenkinsfile Challenges Future
- 3. History Feb 2019 First mentionof AdoptOpenJDK Quality Assurance (AQA) July 2018 Open-source Test Results Summary Service (TRSS) April 2017 Open-source tests to AdoptOpenJDK Sept 2017 Open-source functional tests to OpenJ9 Jan 2018 Initial commit of test Jenkinsfile Feb 2016 Initial plan to “Simplify Java Testing” at IBM Aug 2016 Initial MVP of testkitgen Sept 2016 Outline open- source test strategy & plan for Eclipse OMR Mar 2016 Open-source functional tests to Eclipse OMR July 2019 AQA v1.0
- 4. Many Projects &Servers Language-agnostic (the lower the tests on the stack, the less testing required) JIT IL injection 500+ opcode tests, CTD GC file-based tests leverage verbose GC logs, data from customers Open tools: Googletest framework AdoptOpenJDKOMR OpenJ9 Test infrastructure to wrap all types of open tests Multi-version support (8+) Multi-test support, more than OpenJDK regression tests, FV, SV, Perf, etc Intentionally thin Open tools: Jenkins, TAP FV tests (JIT, GC, VM, JCL) - migrate from 10+ frameworks to 2 Sanity group = 18,000 testcases, 20min serial execution time Standard approaches (exclude, test results, etc) Open tools: TestNG, Jenkins, TAP IBM learn learn learn Proprietary/customized tools (specific for devs, but inflexible & hard to maintain) Decades worth of testing Plus licenses for special/closed test suites Open tools: Few open examples, lots of customized/proprietary tools
- 5. Scope OpenJ9 Hotspot IBMSAP 8 9 10 11 12 + openjdk functional perf system external RI 13 Implementations Platforms Versions Test Categories osxosx aix win xlinux plinux 6 versions aarch 250000 tests 87,000,000 Tests Plus PR builds, promotion builds and personal builds 58 impl_platforms 6 servers Trainload of tests!
- 6.
- 7. Ecosystem AdoptOpenJDK Quality Assurance(AQA) “Make quality certain to happen” Testing a wide criteria representing actual business requirements to identify binaries ready for production usage Today Roadmap (open) Functional correctness Functional correctness OpenJDK regression (open) OpenJDK regression Oracle JCK (closed) Eclipse functional Application & framework tests Builder-specific testing (unknown) Security Passes known vulnerability tests Performance Published metrics Achieves minimum throughput scores Scalability & durability Load & stress testing
- 8. 3-part automation story testkitgenTRSSJenkins Responsibilities: - Categorize tests logically - Generates test targets (level/platform/version/i mpl specific) - Test addition via folders - Parameters: BUILD_LIST, EXTRA_OPTIONS, JVM_OPTIONS, TEST_FLAG, ITERATION Responsibilities: - Schedule regular builds - Provide multiple nodes of all platforms - Parallelization - GUI view of test results Responsibilities: - Monitor multiple servers - Aggregate summary - Deep history - Search/sort/filter - Graphs - Pluggable parsers
- 9. Design Principles IntentionallySimple Reduce friction (less dependencies, less files and locations requiring change) Restrict new features to only those that drive next actions Learn from Past Lessons Granular/flexible - able to run all/some/single tests, tagging to slice’n’dice Execution separate from reporting, able to decouple from CI server Open-source tools when possible Whole team involvement
- 10.
- 11.
- 12.
- 13.
- 14. User Scenarios Localruns (triggered via developer cmdline) PR builds (triggered via PR plugin) Nightly testing (triggered via upstream) Weekly tests (triggered via schedule) Promotion build (triggered via changes && schedule) Grinders (triggered via API, customized, upstream)
- 15. Revision Highlights Initialversion: 1 platform, xlinux SDK resource: upstream Label support & other platforms Create Grinder & merge into single Jenkinsfile SDK resource: nightly, release, customized Iterations & Custom targets Repos/branches params Pre-staged jars (test.getDependency job) Artifactory support Test parallelization Auto-gen jobs (partial and/or full)
- 16. Jenkins Plugins Keep itsimple, limit reliance: Pipeline plugin (now part of core Jenkins) Test results plugins: TAP, Junit Node & Label parameter Xvfb (for tests that require DISPLAY var set) Timestamper Artifactory Github PR builder (for PR build/test runs)
- 17. Challenges Feature creep Ownership & access Technical ego Naysayers Open tools churn (updates, defects, skew) Unstable machine layer Human Technical
- 18. Future Specialized Grinders less parameters in each less confusion for developers Personal Jenkins instances with Docker nodes as additional option for local running of tests Plug in to other CI/CD options Other test-related services Pipeline feeds deep learning
- 19.
- 20. Community building! Pleasejoin us! AdoptOpenJDK adoptopenjdk.net AdoptOpenJDK/openjdk-tests @adoptopenjdk eclipse.org/openj9 eclipse/openj9 @openj9 eclipse.org/omr eclipse.org/omr @eclipseomr Website Github Twitter 8thdaytesting.com smlambert @ShelleyMLambert
- 21. Services to SupportAutomation ResultAnalytics Data Services UI Layer Cores raw refined custom dashboardother clients TestGeneration BenchEngineCoreAnalytics TestSelection BugPrediction InputOptions ResultSummary ResultCompare github repos Jenkins servers DL service
- 22. visualize &analyze data from cores predict crashes based on data mined from core files Core Analytics
- 23. Scores perfile based on ‘recent’ changes due to defects (github PRs/issues), predict based on change & defect history, other features? Bug Prediction
Editor's Notes
- #2 Story about optimism and hard work Heavy train stuck in the mountains, calls for help by big strong engines, each declines… finally train asks for help from this little switch engine… A little railroad engine was employed about a station yard for such work as it was built for, pulling a few cars on and off the switches. One morning it was waiting for the next call when a long train of freight-cars asked a large engine in the roundhouse to take it over the hill. "I can't; that is too much a pull for me", said the great engine built for hard work. Then the train asked another engine, and another, only to hear excuses and be refused. In desperation, the train asked the little switch engine to draw it up the grade and down the other side. “I think I can”, puffed the little locomotive, and put itself in front of the great heavy train. As it went on the little engine kept bravely puffing faster and faster, “I think I can, I think I can, I think I can”. As it neared the top of the grade, which had so discouraged the larger engines, it went more slowly. However, it still kept saying, "I—think—I—can, I—think—I—can." It reached the top by drawing on bravery and then went on down the grade, congratulating itself by saying, "I thought I could, I thought I could." Through a solitary Jenkinsfile, we run millions of tests in daily builds, personal builds, parallel builds against different Java versions, implementations, and platforms. This is the story of our experience leveraging continuous integration tools, iteratively simplifying our initial approach, into a solution that spans across many different servers, user scenarios, challenges and demands. It is the journey of learning and humble revision over the course of a year.
- #3 Through a solitary Jenkinsfile, we run millions of tests in daily builds, personal builds, parallel builds against different Java versions, implementations, and platforms. This is the story of our experience leveraging continuous integration tools, iteratively simplifying our initial approach, into a solution that spans across many different servers, user scenarios, challenges and demands. It is the journey of learning and humble revision over the course of a year.
- #4 - Team of 4, effort to reorganize, share, build and run tests in a simpler, more open and agile way to make our lives inside of IBM easier… Separate execution from reporting - able to run both on dev laptop outside proprietary tools and in CI server Use widely adopted open-source tools, minimize proprietary approaches Granular - able to run all/some/single tests, tagging to slice’n’dice Volume control for test output (logging, discard green results, minimize noise) Reduce friction (less dependencies & locations requiring change) Run RI for comparison and easier triage Incremental improvements, MVPs & identify backlog items that can wait
- #6 10 PRs per day 522, 000, 000 tests/day
- #8 Open testing roadmap becomes AQA
- #11 15 contributors, 150 edits
- #16 TODO – make a timeline (and bigger font) Diagram of all Jenkins server - & all repos in play
- #18 Trolls & naysayers
- #22 In the past we have done various prototypes, we had ideas of services we would like to have, some of them are now in full use, some still parked… With the advent of deep learning, its an appropriate time to revisit these and think about other things that are made more feasible with DL