Amis conference soa deployment. the dirty tricks using bamboo, nexus and xl deploy
•Download as PPTX, PDF•
1 like•1,181 views
This document summarizes a presentation given by Maarten Smeets, András Hetényi and Robbrecht van Amerongen on deployment automation using Oracle Fusion Middleware. It introduces the speakers and their backgrounds. It then discusses challenges with traditional Oracle deployment processes and how gradual automation can help. Key aspects of the Atlassian Bamboo continuous integration tool and how it is used for building and deploying Oracle SOA applications are summarized.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to Amis conference soa deployment. the dirty tricks using bamboo, nexus and xl deploy
Similar to Amis conference soa deployment. the dirty tricks using bamboo, nexus and xl deploy (20)
More from Getting value from IoT, Integration and Data Analytics
More from Getting value from IoT, Integration and Data Analytics (20)
Recently uploaded
Recently uploaded (20)
Amis conference soa deployment. the dirty tricks using bamboo, nexus and xl deploy
- 1. OGh Oracle Fusion Middleware Experience 2016 bij FIGI Zeist Maarten Smeets, András Hetényi and Robbrecht van Amerongen, June-2016
- 3. Who are we? • Maarten Smeets – Senior Integration Consultant – Experience with Oracle SOA Suite since 2007 – Well certified (SOA, BPM, Java, SQL, PL/SQL) – Author of more than 100 blog articles https://nl.linkedin.com/in/smeetsm http://javaoraclesoa.blogspot.com maarten.smeets@amis.nl • Robbrecht van Amerongen – Manager Continuous Delivery – Initiator of the Red Expert Alliance – Agile Master www.linkedin.com/in/robbrecht +31641010286 robbrecht@amis.nl • András Hetényi – Continuous Delivery and Integration Consultant – Senior Oracle (SOA) developer https://www.linkedin.com/in/kleopardo andras.hetenyi@amis.nl
- 4. 4
- 5. 5
- 6. Enterprise SOA deployment means
- 7. How does it feel
- 8. People who break stuff “developers” “ops team” People who deal with the consequences Talk about DevOps
- 9. People who break stuff less and deal with the consequences “Devops team”
- 10. Non – technical tricks • Start small Show it in real software. Hijack a server and run a build engine • Show the build dashboard on a TV screen So you get questions from management and gain acceptance • Bribe your server admin Getthe admin on board and buy sweets, “real” coffee / tea
- 11. Non – technical tricks • Tool is fit for purpose Use specific tools for specific solution • Include legacy Work with legacy, not around it. Show it is working for all sorts of products • Make the project the owner of the tools first. Next find an organizational owner for Build en Deployment tools. Take care of lifecycle management of the tools • Communicate, communicate, communicate Project newsletter. Updates on the corporate intranet. Show it is there.
- 12. Acceptance The Continuous Delivery Pipeline For SOA Scrum team System 2 Scrum team System 1 Scrum team System n SVC Automated Provisioning omgeving Build Domain Specific Language Continuous Integration QA Automated deployments / testing Develop OS and Middleware DEV Test OS and Middleware Integration OS and Middleware ACCEPT OS and Middleware Production OS and Middleware PROD ArtifactRepository WebLogic Scripting Tool (WLST)
- 13. What to build / deploy • An artifact – Is deployed to a single environment – E.g. SOA, Service Bus, configuration, ADF – Configuration is not part of a service! • A functional unit (deployable unit) – Is the unit of version control – Always deployed as a single entity – Is a collection of artifacts – Is the input for the deployment process • A release – Is a collection of functional units – Has a specific version – Is deployed to acceptance and production – Is the complete set of software Artifact Functional Unit Release WebLogicScriptingTool(WLST)
- 14. Why Atlassian Bamboo? • Experience / implementations at different customers • The tool and vendor are regarded as best of breed-class • It is very strict by design, which is useful for sticking to the pipeline proces • From Atlassian, easily linked with Jira and Confluence • It is cheap AND has full support from Atlassian Tip: Scripts to bridge the gap between build server and CD platform should be CD tool independent!
- 16. Bamboo How does it look
- 17. Bamboo setup • Why no Bamboo deployment project? – Requires specifying a branch. Bamboo branches are used to specify a functional unit. Would require a deployment project per functional unit – A Bamboo release is a snapshot of a number of Bamboo artifacts Bamboo is not used as artifact store What would be the artifact? Which branch?
- 18. What to use for build / deploy scripts • WLST (essential part of WebLogic server) – Flexible, can be extended with Python modules and Java libs – Often used by deployment tools / plugins – By default uses Jython 2.2.1 (2007) • Ant (10g, 11g, still there in 12c) – Flexible – Poor dependency management can be fixed with Apache Ivy – Can become complicated • Maven (12c and later) – Not meant for scripting – Requires plugins WebLogic Scripting Tool (WLST)
- 19. Challenges building with Oracle and Maven / ANT • Scripts from hell – That one person who knows how it works – Tight coupling between scripts and deployment tooling (migration will be painful) • The incremental database and Maven – Cultural issues: it has to be done with SQLPlus – Technical issues: I want to be able to rebuild my database from scratch to create a new environment. I want to combine the increments of several sprints • Maven only for 12c+ – Maven for all artifact types? Rerunnable code (within sprintrelease)! Store database code per sprintrelease. Store the incremental set of sprintreleases to recreate an environment. Never do manual changes! KISS. Harder than you think! Standardize. Modular. Reduce the number of different products and component types. Share responsibility Keep all your code in the artifact repository. A release is just a list of artifacts
- 20. Artifact repository A single source of truth • Immutable – Once an artifact with a specific version is in the artifact repository, it does not change. A specific version is always the same code. • Traceability – Artifact to version control to build tooling traceability You can use the version for that. e.g. 1.0.[revision].[build number] • Build once deploy many – Apply the ‘Build once, deploy many’ pattern. Compilation does not always yield the same result • Proxy dependencies: Oracle Public Maven repository – Proxy the Oracle public Maven repository • Tip: Environments change. Do not store deployment plans / configuration plans with hardcoded endpoints!
- 21. Artifact repository Structure Functional unit Artifacts Versions Oracle Maven Repository is proxied
- 22. SOA testing • Unit tests – Tests the isolated unit. Dependencies are mocked – Data services (e.g. database) are not mocked • Functional tests – Tests the service including interactions with dependencies – No mocks • Tip: Tests should be independent of data. Test first prepares data, executes test, removes data Mocks
- 23. Deployment SOA Loose coupling • Dependencies create complexity – Dependency loops – Installation ordering in build code required – Can cause server start issues Avoid direct service artifact dependencies! • Use MDS for shared artifacts • Do not use SOA-Direct Service A Service B Cyclic dependency https://blogs.oracle.com/aia/entry/aia_11g_best_practices_for_dec
- 24. SOA code quality
- 25. SOA code quality • SonarQube ojaudit plugin available, but no SOA audit rule-set (requires Java coding) • Code Compliancy Inspector (CCI) from AIA https://docs.oracle.com/cd/E28280_01/doc.1111/e15754/codecomplianceinsp.htm#OERIN877 https://github.com/wvanderdeijl/oracle-cci-sonarqube • Payload validation should be turned on in development / test! Do we have a set of SOA development standards we can agree on?
- 27. OPS DEV Legacy Deployment Process Delivery DEV WLST SQL*Plus Unix shell TST INT FAT UAT PRD Sources Production releases about per quarter vs. bi-weekly sprint deliveries from the project teams
- 28. OPS DEV Gradual Automation DEV TST INT FAT UAT PRD WLST SQL*Plus Unix shell Production releases are every two weeks following the sprint deliveries from the project teams Sources
- 30. User Interfaces • Command Line Interface (CLI): – Jython application that the user can access remotely. • GUI – Browser client requiring Flash-plugin – Configure and perform deployments – View the release pipeline of the different applications – View and edit the repository – View reports – View or edit security settings (based on authorization)• REST API – Basic authorization – Can be accessed by URL http[s]://<xld_host>:<xld_port>/deployit/<Service> – All public services are exposed: • E.g.: Control, Server, User, Repository, Package, Deployment… – Example: • curl -i -X GET -H "Authorization:Basic YWRtaW46YWRtaW4xMjM=" ‘http://localhost:4516/deployit/repository/ci/Infrastructure/Lo cal/LocalWL’
- 32. Packaging • Creating a deployment package: – With the GUI: • Directly by creating items under an application version • Can be exported to DAR – With the CLI: • Through Jython objects representing the REST API services – With the XLDeploy Plugin for DAR Art1v1.0 Art3v1.2 Art4v1.1 manifest APP/Version deployables specification1 + URI1 specification2 specification3 + URI3 specification4 + URI4 Art1v1.0 Art3v1.2 Art4v1.1 POM Project/Version deployables specification1 + URI1 specification2 specification3 + URI3 specification4 + URI4
- 33. CI-s, types and properties Deployable: soa.CompositeSpec Deployed: soa.Composite Container: wls.Container wls.Container: wls.Domain wls.Cluster wls.Server
- 36. – Specification: • Which deployables to which containers • How to configure them – Delta analysis: • Specification vs. current state of middleware • Results in Delta specification: – CREATE: First time deployment – MODIFY: Upgrading the item – DESTROY: Undeploy the item – NOOP: No change – Orchestration: • Delta spec into sub-specs, executed in isolation • Resuls in Deployment plan with sub-plans – Planning: • Adding Steps to each sub-plan needed to execute the deployment • Example: database restore point – Execute: • Execution of the complete deployment plan XL Deploy at Work
- 37. Mapping and Plan
- 38. Migration Scenarios • Adaptation to XL Deploy – Extend with DAR generation – Re-define the artifacts – Covered by the XL Deploy plug-ins – Significant effort from developers – XebiaLabs support applies • Adaptation to Legacy – Extend with DAR generation – Keep the artifacts except for placeholders – Development in XL Deploy – Minimal effort from developers – Not supported with the product, must be re-validated for each update
- 39. Generic SOA-OSB Services MDS BPM OSB Shared G.O. Data sources DB Adapter config JMS (CF,SRV,MDL,Q,T) config plan customization config plan config plan customization SCA SOA-OSB Artifacts SCA MDS BPM OSB Shared G.O. config plan config plan customization Manifest DS DBA CF DBA DS DS DS DBA DBA Q T Adaptation of Legacy to XL Deploy Artifact migration
- 40. Adaptation of XLDeploy to Legacy (Example of generic database scripts) Deployable dbconfig |– init.sql |– insert_refdata.sql |– insert_userdata.sql |– create_indexes.sql |– create_sequences.sql |– create_tables.sql |– grants | |– privileges.sql | |– roles.sql |– logging | |– loginit.sql | |– log_functions.sql |– patch.sql |– setup.sql |– update_packages.sql Deployable dbconfig |– 00-setup | |– insert_refdata.sql | |– insert_userdata.sql |– 00-setup.sql |– 01-patch | |– create_indexes.sql | |– create_sequences.sql | |– create_tables.sql | |– privileges.sql | |– roles.sql | |– update_packages.sql |– 01-patch.sql |– common | |– loginit.sql | |– log_functions.sql Wrapper installdb.sh db_log.sh Plugin custom.DbScript property: listOfScripts setup.sql patch.sql Deployable dbconfig |– init.sql |– insert_refdata.sql |– insert_userdata.sql |– create_indexes.sql |– create_sequences.sql |– create_tables.sql |– grants | |– privileges.sql | |– roles.sql |– logging | |– loginit.sql | |– log_functions.sql | |– patch.sql |– setup.sql |– update_packages.sql Plugin installdb_xld.sh db_log_xld.sh Plugin installdb_xld.sh db_log_xld.sh v.s.
- 41. Filling the Gaps: Rollback CREATE File Folder Ear War Package DESTROY File Folder Ear War DDL Package CREATE SOA BPM Package MODIFY 1.1-1.2 MODIFY 1.2-1.1 DESTROY Default
- 42. Orchestrating application components by custom rules. Example: Interrupt + Rules Filling the Gaps: Dependency Execute PRE Deploy SOA-OSB Execute POST SOA-OSB-Config Scripts PRE POST PAUSE SOA-OSB Services
- 43. 44
- 44. 45
Editor's Notes
- This is a familiar image to MN
- This is a familiar image to MN
- Tip: Configuration (e.g. datasources, JMS) should not be part of a service functional unit since there is only a single version of configuration per release. Determining the configuration per release is more difficult as it is spread out over services Tip: Less types of deployables makes it easier for everyone Pom -> DAR
- Changes in artifact structure are troublesome. 12.2.1 libraries not yet in Oracle Maven public repository. OSB 12.2.1 and Maven is troublesome
- Usually we show up with a new tool for an existing legacy solution otherwise it would be much easier. What we can see here? Developer teams committing their code to the version control, testing and integration runs properly the agile way. The build process is able to generate the releases. On the other side, operations team work with a combination of manual steps, cloned artifacts, configuration and deployment scripts. The build process in Bamboo provides the Nexus repository with most of the artifacts but the deployment process is using other location to pick them up.
- The desired result of automation looks like this. Nexus is the single repository for all deployables and their dependencies We get rid of all kind of copying through the chain Any kind of formerly manual or losely scripted action gets integrated into the XL Deploy tool. This goal may lead to unrealistic expectations. Like deployment team believes they push one button and after 2 hours they see Done.
- XL Deploy is an agentless application release automation tool across all platforms. It is running as a Java process Operates with a JCR repository and LDAP security Interacts with its destinations through technology plugins, which are loaded at startup CORE Integrates with plugins Unified Deployment Engine determines deployment requirements Storage and retrieval of deployment packages Executes deployment plans Security Reporting PLUGIN Interaction to a middleware Extendable Core requests, plugin replies All loaded at startup Licensed plugins: come with the installation, supported Community plugins: open source (GitHub), not supported
- The fundamental layer of the user interface is the REST API. Depending on the security requirements it is accessed via http or https. It uses basic authentication and exposing pre-defined service objects for the different phases of the deployment process. For instance server, user and control for maintenance, monitoring and security management, repository to maintain the CI-s and deployment and task to create plans and interact with the deployment process. The other 2 interfaces are built upon this REST API. Command Line Interface is a separate installation of Jython libraries for remote communication with the XLD server. The Graphical Interface is a browser client works with Flash plugin. It visualizes most of the exposed services, furthermore, plugins can also expose their functionality using Java annotations
- Package: A version of an application Deployables: Configuration Items (CIs) that can be deployed: Artifact: Physical file in the repository Resource: Created on the target platform e.g. JMS queue/topic, DSource, CF Containers: Deployment targets Domain, server, cluster, DB client, Local, SSH, CIFS Dictionary: Collection of placeholders and their values Environment: Collection of references to infrastructure CI-s (members) Blue: CREATE Grey: NOOP Green: MODIFY Red: DESTROY
- Let's see how we feed the XL Deploy repository. In the Maven project we define the resources with their properties and when applicable, the deployable artifact in the form of standard dependency (groupId, artifactId, version, type and optional classifier). The XL Deploy plugin wraps the resource list with the package definition forming the manifest file, downloads the dependency artifacts from the Nexus repository and packs them all together into a DAR file.
- Here is an example how the basic Oracle SOA plugin treats its artifacts. This plugin is in practice a derivation from WebLogic plugin. It uses new artifact types, enriched with properties that SOA Suite deployment requires. The target containers, however, are of the same types as for the WebLogic deployments. How to prevent the mapper to spread the composite all over other non-SOA clusters? For example with tagging.
- Here is an example of the combination of plugin type system and the tagging feature. The SOA Suite plugin extends the WebLogic plugin with the artifact types soa.DbConnectionFactory and soa.Composite and associates them to the container wls.Cluster. Since the container type may represent other weblogic cluster as well, we have to reduce the candidates to our SOA domain. We stick a label to each artifact and put both labels on the tag list of the cluster.
- As we saw, the XLD environment is a collection of references to potential physical containers and placeholder dictionaries. The deployment process this way can in the deployed artifact combine the properties of the deployable, the replaced values using the dictionaries but also attributes of its actual container.
- The deployment process in time is a good co-operation between the core process and the plugins. The plugins determine the source and the destination through their types and the tags Delta analysis: Important to note, current state of middleware virtual image in XL Deploy! Orchestrators: by container, by deployment group, sequencing by dependency
- Each deployment action starts with the selection of a deployment package and its destination from the environments. If no version of the same application is deployed there, the engine recognizes it as an initial deployment otherwise as an update. In this example the package contains a SOA composite only. The review shows us all steps generated by the plugin. Reviewing the main deployment step we can see several contributors to the final plan. The order of the execution within the plan is configurable, the name of the actual rule (rules can be added, disabled and customized) and the script (in this case meant for WLST) template from the SOA plugin being applied on the target. The actual result is however a wrapped script containing initializations and supporting functions from another plugin for python.
- Vervolgens 2 voorbeelden
- The all-in archive of a service is replaced by the elementary units The environment specific parts (config and customization) are replaced by their placeholdered templates The scripted objects are replaced by manifest items without file artifacts
- At the customer we confronted with complex database script structures surrounded with a logging mechanism within the database. Keep the encapsulated object Trust all internal orchestration to the deployable (=developers)
- Considering rollback scenarios is one of the critical points of deployment. Simple files or folders are usually easy decision. We can test it out very well what to do to recover after deleting or overwriting too much. Not necessarily a rollback, more to keep data integrity. Assuming an update when we create a new database table and filling it up with critical data or new column with constraints etc. After some runtime we downgrade the containing release but a rollback would not succeed or would cause inconsistency. Automating an initial restore point and reverting to it at the end may help.
- The last example is related to the efficient encapsulation of a legacy process. The deployable objects are structured folder with configuration scripts and the property lists. The same scripts take care of configuration before and after a SOA-OSB service deployment, determined with different arguments PRE and POST. For delivery reasons, the configuration script got its own package. According to the previously discussed topics, the legacy object is already prepared with placeholders and tags. The standard plugin would just extract the folder to the target host with one step and that's it. To organize the proper sequence, we can define a new rule and associate to our artifact. The new rule will surround the extraction with 3 extra steps: 1. Execute the main script with PRE 2. Suspend the plan 3. Execute the main script with POST Between 2 and 3 we execute the SOA-OSB deployment task. Then we resume the waiting task.