Instantaneous Replication of Build Artifacts with NetApp
•Download as PPTX, PDF•
1 like•725 views
Did you miss our session on instantaneous replication at #swampUP 2017? Catch up on everything you may have missed.
More Related Content
What's hot
What's hot (20)
Similar to Instantaneous Replication of Build Artifacts with NetApp
Similar to Instantaneous Replication of Build Artifacts with NetApp (20)
More from NetApp
More from NetApp (20)
Recently uploaded
Recently uploaded (20)
Instantaneous Replication of Build Artifacts with NetApp
- 1. www.swampup.jfrog.comCopyright © 2017 JFrog. All Rights Reserved Instantaneous replication of build artifacts with NetApp Bikash Roy Choudhury, Principal Architect Edmilson Peixoto, Solutions Development
- 2. © 2017 NetApp, Inc. All rights reserved. --- NETAPP CONFIDENTIAL ---2 Bikash Roy Choudhury NetApp Principal Architect California Bikash Roy Choudhury is a Principal Architect at NetApp. He is responsible for designing and architecting solutions for DevOps workflows relevant across industry verticals including high tech, financial services, gaming, social media, and web-based development organizations that address customer business requirements in these markets. He also works on integrating and validating solutions with Red Hat (RHOSP- IaaS), Apprenda and IBM Bluemix (PaaS), Docker Containers, CloudBees Jenkins, Jfrog, Kubernets and Perforce Helix using RESTful APIs and integrating them with NetApp ONTAP software in private, hybrid, and public clouds.
- 3. © 2017 NetApp, Inc. All rights reserved. --- NETAPP CONFIDENTIAL ---3 Edmilson Peixoto JFrog - US Dir Solutions Development Ed Peixoto is the Solution Development director at JFrog. He is responsible for leading the development of customer and partnership integration & solutions enabling the connection of JFrog products to other DevOps companies and technologies. Before joining JFrog, he worked for 16 years at Ericsson as software engineer (R&D) and as head of the Development Environment CI/CD organization.
- 4. © 2017 NetApp, Inc. All rights reserved. --- NETAPP CONFIDENTIAL ---4 Distributed software development has become common
- 5. © 2017 NetApp, Inc. All rights reserved. --- NETAPP CONFIDENTIAL ---5 Ensure all developers are using the same versions
- 6. © 2017 NetApp, Inc. All rights reserved. --- NETAPP CONFIDENTIAL ---6 Ensure all build artifacts are shared efficiently with global teams
- 7. © 2017 NetApp, Inc. All rights reserved. --- NETAPP CONFIDENTIAL ---7 Reduce risk of network latency and stability
- 8. © 2017 NetApp, Inc. All rights reserved. --- NETAPP CONFIDENTIAL ---8 Access any specific versions on remote artifacts
- 9. © 2017 NetApp, Inc. All rights reserved. --- NETAPP CONFIDENTIAL ---9
- 10. Key DevOps Distributed Data Challenges • Managing, sync’ing, and sharing data and code across multiple development locations – often globally • Controlling the communication costs of distributing data between multiple locations • Supporting and managing heterogeneous data • Protecting and securing distributed data repositories • Adapting data management practices to 3rd platform development environments • De-risking DevOps and ensuring Governance and Compliance requirements are met © 2017 NetApp, Inc. All rights reserved.10
- 11. Where do JFrog and NetApp play in Eco-system? © 2017 NetApp, Inc. All rights reserved.11 Workflow Workload Data Management Shared File Services Simple Storage Service (S3) (Object) Data Access Data Control CI/CD
- 12. Global Replication & Collaboration with StorageGRID Webscale User ingests data in San Francisco User reads data in TokyoStorageGRID Site 1 StorageGRID Site 2 StorageGRID Site 3 StorageGRID Site 4 Data Placement – Policy Driven 70PB Capacity 100B Objects 16 Sites © 2017 NetApp, Inc. All rights reserved.12
- 13. JFrog-StorageGRID Replication and Collaboration Solution details © 2017 NetApp, Inc. All rights reserved. --- NETAPP CONFIDENTIAL ---13
- 14. Use Cases • DevOps • Block Chain • IoT Shared Data for Distributed systems © 2017 NetApp, Inc. All rights reserved. --- NETAPP CONFIDENTIAL ---14
- 15. Why JFrog Artifactory on NetApp? Provides scalability and performance for build repositories and artifacts in the CI pipeline Instant metadata updates between remote sites with JFrog database replication Read/write copies of files are downloadable almost instantly for local users at each site (wire speed is the limiter) Disaster recovery from accidental file deletions and site disasters are quick and easy Reduced cost with StorageGRID compared to CRR offering from a popular hyperscalar. JFrog Artifactory is used as a universal artifact repository management tool
- 16. Site Replication overview Application vs. Array replication © 2017 NetApp, Inc. All rights reserved. --- NETAPP CONFIDENTIAL ---16 Site 1 Site 2 Application level Replication Metadata Array level Replication Data Application aware Data Replication Metadata + Data
- 17. Business Problems to solve • Performance • Replicating files to remote sites across WAN • Collaboration • Instant availability of read/write copy of the build artifacts in all remote locations with limited communication cost overhead • Data Protection • High volume of build artifacts can be stored, archived and managed cost effectively JFrog with NetApp StorageGRID Webscale © 2017 NetApp, Inc. All rights reserved. --- NETAPP CONFIDENTIAL ---17
- 18. Archive and Share Build Artifacts across Remote Sites JFrog DB and checksum repository on AFF with ONTAP 9; StorageGRID for multi-site replication and DR © 2017 NetApp, Inc. All rights reserved. AFF Builds Artifacts S3NFS Database AFF Builds Artifacts S3 NFS Database Site 1 Site 2 Instant Replication (PUSH) (PUSH) Artifactory MySQLArtifactoryMySQL
- 19. Multi Site Collaboration with Distributed Content Repository Syncing up StorageGRID deployments at all locations © 2017 NetApp, Inc. All rights reserved. USA Asia Europe Australia Database Build ArtifactsDatabase Database Database JFrog Sync the DB JFrog Sync the DB Global Namespace (Instant Replication) Global Namespace (Instant Replication) Build Artifacts Build Artifacts Build Artifacts
- 20. Cross Site Replication Performance Results Popular Hyperscalar Cross Region Replication (CRR) versus NetApp StorageGRID Webscale © 2017 NetApp, Inc. All rights reserved. Environment Details 100 artifacts 95MB size Push to Amazon S3 and NetApp StorageGRID Webscale Replicate across two sites WAN latency is 170ms 0 5 10 15 20 25 30 Popular Hyperscalar CRR NetApp StorageGRID Series1 4.74 26.42 ThroughputinMB/sec Artifact Replication using Jfrog Artifactory and NetApp StorageGRID 6x performance improvement
- 21. NetApp-JFrog Value Modernize Your Infrastructure (Application is transparent of the data format) Bi-directional placement of artifact & build access over files & Objects Data Continuity (App aware data driven) Accelerate Your Applications (Faster data replication and collaboration) Data Access and Control (Instant file listing) Instant metadata replication System Stability and reliability with HA Data Protection and Security (Erasure Coding and Storage Efficiency) Transform Data Protection (Protect Data in every site) NetApp Insight © 2016 NetApp, Inc. All rights reserved. NetApp Confidential – Limited Use Only21
- 22. Thank You. © 2017 NetApp, Inc. All rights reserved.
- 23. Continuous Integration (CI) Framework Different modules in the CI framework using ONTAP APIs and mounted on Docker containers © 2016 NetApp, Inc. All rights reserved. NetApp Highly Confidential Information. Unreleased product plans may change. Not for distribution. Source Code Repository • Source Code version Control • No pre-build artifacts Continuous Integration (CI) • Integrate builds • CI tests Developer Workspaces • Prepackaged Workspaces • Private builds Build Artifacts Repository • Resolve Build Dependencies • Manage Build repositories
Editor's Notes
- Distributed software development has become commonplace, especially in large enterprises that have several sites in different locations around the globe. This presents many challenges to ensure that all the development teams work on a coherent and synchronized code base. Here are things we need to guarantee: - Ensure all developers are using the same versions - Ensure all build artifacts shared efficiently with global teams - Reduce risk of network latency and stability - Access any specific versions on remote artifacts To overcome these challenges, Artifactory supports several ways to replicate repositories and accommodate the distributed topology that best fits your business needs. We had a use case in which a customer required fast propagation of new build artifacts and constant collaboration across four geographically separated sites
- - Ensure all developers are using the same versions - Ensure all build artifacts shared efficiently with global teams - Reduce risk of network latency and stability - Access any specific versions on remote artifacts
- - Ensure all developers are using the same versions - Ensure all build artifacts shared efficiently with global teams - Reduce risk of network latency and stability - Access any specific versions on remote artifacts
- - Ensure all developers are using the same versions - Ensure all build artifacts shared efficiently with global teams - Reduce risk of network latency and stability - Access any specific versions on remote artifacts
- - Ensure all developers are using the same versions - Ensure all build artifacts shared efficiently with global teams - Reduce risk of network latency and stability - Access any specific versions on remote artifacts
- We have a use case in which a customer requires fast propagation of data and constant collaboration across four geographically separated sites. Data is the common denominator for all JFrog Repositories and NetApp and super fast replication is what we want to deliver in this use case. So, how NetApp can help with in this solution?
- Here is a list of the challenges and pain points that we will discuss. Transformation to either establish or increase a web presence What are the requirements? How do you get it right? Migration from traditional applications to modular and NoSQL technology platforms Also, what do you do about existing data? How can you leverage that existing data with your new analytics platforms? Then as you migrate your use cases to 3rd platform applications, how are you going to protect and manage your data? Choices for enterprise grade data protection are very limited, if they exist at all. Realizing the importance of data protection, it makes sense to go a little deeper on that subject as we discuss Enterprise grade data protection. Finally, the initial configuration of NextGen applications and DB environments is not always clearly defined. The ability to address that requires a level of flexibility that is not necessarily native to 3rd platform applications. ============================================== Sub-optimal ROI on Big Data deployments include discussion of data sprawl Use case description: Traditional brick-n-mortar vendor expanding their online presence Re-architecting from 2nd platform to 3rd platform Challenges: Inability to handle spikes due to online peaks CAPEX challenged and cannot host On-Prem – hence they are scaling into cloud or moving entirely to cloud Dealing with regulatory and compliance requirements Post analytics (data lake- mapreduce, spark is running on HDFS and results get dumped to storage tier)- we could use hybrid FAS – show value of ONTAP hybrid FAS NetApp: provides high capacity storage , perf of flash pool. Aff a200-> A 700 perf->hybrid fas
- ONTAP 9 and All Flash FAS (AFF) improve storage efficiency and provide high uptime Changes to one site are propagated to other remote sites for instant listing Replication with StorageGRID provides 6x performance over Amazon S3 Cross Region Replication (CRR) using JFrog Artifactory StorageGRID stores the same copy of the files in all locations 1c per GB/month with StorageGRID versus 4.5c per GB/month with Amazon S3 CRR Keeping a file in S1 is 2.25c per site. Two sites is 4.5c. Copying between sites is 2c. AWS S3 charges 9c per GB for downloading <10TB
- Single global namespace Instant metadata replication Asynchronous file copy between sites.