Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

From Mainframe to Microservices: Vanguard’s Move to the Cloud - ENT331 - re:Invent 2017

1,247 views

Published on

Maintaining control of sensitive data is critical in the highly regulated financial investments environment that Vanguard operates in. This need for data control complicated Vanguard's move to the cloud. They needed to expand globally to provide a great user experience while at the same time maintaining their mainframe-based backend data architecture. In this session, Vanguard discusses the creative approach they took to decouple their monolithic backend architecture to empower a microservices architecture while maintaining compliance with regulations. They also cover solutions implemented to successfully meet their requirements for security, latency, and end-state consistency.

  • Be the first to comment

From Mainframe to Microservices: Vanguard’s Move to the Cloud - ENT331 - re:Invent 2017

  1. 1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS re:INVENT From Mainframe to Microservices: Vanguard’s Move to the Cloud I l y a E p s h t e y n , A W S S o l u t i o n s A r c h i t e c t B a r r y S h e w a r d , V a n g u a r d C h i e f E n t e r p r i s e A r c h i t e c t E N T 3 3 1 N o v e m b e r 3 0 , 2 0 1 7
  2. 2. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. What to expect from this session Mainframe modernization approaches Vanguard’s cloud data architecture and mainframe strangulation strategy Benefits and lessons learned
  3. 3. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Characteristics of microservices architecture Reusable modules of code that are built and deployed as independent entities Organized around business capabilities Own their domain logic Decentralized governance and data management Automation and designed for failure
  4. 4. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Microservices benefits Removes business and data logic from applications Helps customers reduce technical debt Eliminates monolithic bottlenecks Improves developer velocity
  5. 5. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. monolithic application + monolithic teams But how do you get there from here…
  6. 6. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. • Re-host—emulator-based approach • Reengineer • Refactor (automated)—automatic code conversion and refactoring • Rewrite (manual)—re-developed as microservices • Batch job migration—cloud-native optimized batch processing • Data analytics enablement—cloud-native data lake and analytics • Data-driven augmentation—cloud-native data-based services • Re-platform—Java and Linux migration • Repurchase—code replaced with new packaged solutions • Retire—code abandoned while data is migrated or archived • Combination of the above Mainframe modernization approaches
  7. 7. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Modernization speed to business value Cloud business value 6 months 1 year 2 years 3 years 4 years 5 years Time Rewrite Refactor Re-host Repurchase
  8. 8. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.© 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Martin Fowler’s Strangler Pattern “…gradually create a new system around the edges of the old, letting it grow slowly over several years until the old system is strangled.” Martin Fowler June 29, 2004
  9. 9. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. No New Features on the Monolith Creating a new system around the edges Monolith Browser DB Proxy Browser Service DB DB Monolith Proxy Browser Service DB Service DB Service DB DB Monolith
  10. 10. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. API gateway Browser Service DB Service DB Service DB Monolith DB New system may not always lead to strangulation
  11. 11. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. API gateway Browser Service DB Service DB Service DB Monolith DB A better approach: Focus on the data
  12. 12. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Vanguard’s Journey from Mainframe to Microservices
  13. 13. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Vanguard is one of the world's largest investment companies, offering a large selection of low-cost mutual funds, ETFs, advice, and related services Core purpose—To take a stand for all investors, to treat them fairly, and to give them the best chance for investment success Oldest fund—Wellington Fund (inception 1929) Began Operations—May 1, 1975 in Valley Forge, PA Funds—Over 180 U.S. funds (including variable annuity portfolios) and 190 additional funds in markets outside the United States Vanguard—Background
  14. 14. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Vanguard’s IT Environment • Multiple data centers plus disaster recovery datacenters o Huge reserves to handle spikes • Custom web applications o Monolithic Java applications • Large and complex o Stateful, requiring sticky sessions o Use internal non-distributed cache o Largest Lines of Business (LOBs) use data mainly in DB2 on the mainframe o Other LoBs use other databases Complex and Inflexible Mainframe CacheCache LoB#1LoB#2LoB#3 Web Tier App Tier DB Tier
  15. 15. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. • z/OS (MVS) • Build system for COBOL • Data in DB2 supports online systems o 3,000+ relational tables o 6,000+ COBOL stored procedures • Record-keeping systems o VSAM files o CICS interfaces o COBOL batch processes • Integration logic o MQ o COBOL Vanguard’s Mainframe Environment Complex and Inflexible Mainframe DB2 Record Keeping Integration LogicMQ CICS COBOL SPs
  16. 16. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. From Legacy Web Applications to Microservices • Microservices solve monolith problem o Stateless—horizontally scalable o Strict bounded context via network o Simplifies continuous deployment (CD) o Enables polyglot—multiple types of data store or multiple programming languages. To the Public Cloud • Supports Infrastructure as Code (IaC) • Enables managed services • Makes elasticity viable Off the Mainframe • Mainframe prevents pure public cloud • Cost (hardware and software)—horizontal scaling • Desire for common programming model • Skills availability • Continuous deployment through to data layer Why migrate?
  17. 17. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Next-Gen Applications (NGA) MONOLITHIC APPS CLEAN, MODULAR CODE APPS & SERVICES CLOUD CONTINUOUS DELIVERY / DEVOPS TRUE AGILE / LEAN IT NEXT GEN APPS Isolate Data Layer Bounded Contexts Elasticity Simple and Flexible
  18. 18. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Low latency Compliant Cost optimized Secure Multi-region geo distributed Autonomous within region NGA in the Public Cloud Simple and Flexible
  19. 19. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Initial Approach—Next-Gen Data • Met requirements • Hub and spoke • Cloud-hosted spokes • Incorporated past decisions • Object relational mapping • Business events • Two-phase commit • Asynchronous • Proprietary hardware • No managed platforms
  20. 20. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Simplified Cloud Data Architecture • Supports: o CDC to DB o Read from DB o Direct write • Easy improvements: o DB HA • Multi-AZ o DB scalability Option 1 μservice CDC CDC μservice ExtendedHub
  21. 21. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. CDA—RDS with Buffered Writes Option Buffered Writes + User-perceived latency + Availability + Disaster impact + Smoothens spikes - No single integration point - Asynchronous writes Option 3 μservice CDC CDC μservice Replicator Dispatcher DB Writers ExtendedHub
  22. 22. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. • Three separate AWS Kinesis streams: o To-Do o Done o Error • Helps support synchronous-like operations • Provides full situational awareness • Kitty Hawk o Deals with in-flight messages during write operations! Buffered Write Deep Dive μservice μservice Replicator Dispatcher DB Writers Replicators Kitty Hawk μservice Kitty Hawk μservice To-Do Stream Error Stream Done Stream
  23. 23. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. CDA—NoSQL Support Option Features: + NoSQL data store + Relational to NoSQL mapping + Single point of integration - Slightly higher latency - Complexity re- introduced Option 6 μservice CDC CDC μservice Replicator Dispatcher DB Writers Event Writers ExtendedHub
  24. 24. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Cloud Data Architecture—Final Features: + RDBMS databases + NoSQL DynamoDB databases + Single point of integration + Meets availability requirements - Complexity re- introduced μservice CDC CDC μservice Replicator Dispatcher DB Writers Event Writers Big Data ExtendedHub
  25. 25. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Mainframe Strangulation Strategy • Replicate data to the cloud • Refactor the monolithic apps so web pages make AJAX calls to microservices • Migrate batch processes to the cloud, using data in the cloud • Gold copy in cloud—Reverse replication • Treat mainframe record keeping systems as bounded contexts o Integrate with them from the cloud
  26. 26. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Relational Database Service (Amazon RDS) SOC2 Compliance ✓ Data at Rest Encryption ✓ User Access Management ✗ Home-grown IDM interface Data Activity Monitoring ✗ Home-grown DB log monitoring
  27. 27. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon DynamoDB SOC2 Compliance ✓ Data at Rest Encryption ✗ Client-side encryption User Access Management ✓ Data Activity Monitoring ✗ (Write- Only) DynamoDB Streams Client-side encryption + KMS & CloudTrail
  28. 28. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. AWS Lambda SOC2 Compliance ✓ App selection AWS engagement Data at Rest Encryption n/a User Access Management ✓ Data Activity Monitoring n/a
  29. 29. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Kinesis SOC2 Compliance ✓ App selection AWS engagement Data at Rest Encryption ✓ User Access Management ✓ Data Activity Monitoring ✗ Client-side encryption + KMS & CloudTrail
  30. 30. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Microservices + Continuous deployment (CD) pipeline means: o Pull request model forces peer review o Quality gates o Rapid feedback + Microservices principles means: o Strictly enforced bounded contexts o Stateless - Eventual consistency Other + Single development model + Polyglot—data stores, language • Compliance Cloud + Enables infrastructure as code (IaC) + Continuous deployment pipeline means: + Pull request model forces peer review + Quality gates + Rapid feedback + Managed services allow focus on the business problem, not the infrastructure - Eventual consistency / latency Impacts of Migration
  31. 31. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Regulatory • Be aware of regulatory compliance needs, particularly scope, and process for risk acceptance rather than risk avoidance. • Understand data classifications for different bounded contexts. Data with certain needs can be treated differently, for example integrity rather than confidentiality. • Have a backup plan in case a primary choice isn’t available due to compliance or functionality delivery issues. Often a self-managed solution is available. Acceptance • Publicize the architecture to managers and engineers. Make them part of the solution. Prepare for pushback. • Work with smaller teams to understand problems and gain consensus. Larger teams for approvals. Cloud Specific • Be prepared to re-architect as AWS releases new features and defer decisions whenever possible. • Build a relationship with the AWS team. Your Technical Account Manager and Solutions Architects. o Let AWS know if there is a gap in their feature set that is needed. You may find out that the capability is under development and you can beta-test it. Lessons Learned
  32. 32. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Thank you!

×