Recommended
More Related Content
What's hot
What's hot (20)
Similar to Managing Microservices with Neo4j
Similar to Managing Microservices with Neo4j (20)
Recently uploaded
Recently uploaded (20)
Managing Microservices with Neo4j
- 1. Managing Microservices with Neo4j Rob Schoening Ashley Sun
- 2. Overview • Intro • MacGyver + Neo4j • Use Cases • Q&A
- 3. • World’s largest online credit marketplace – Peer-to-peer personal loans – Small business loans – Medical financing
- 4. Micro Services All the cool kids are doing it… …and it makes complex software easier to write and maintain
- 5. A long, long time ago… 2 years ago… We ran 5 production services… Now we run… 139
- 6. That’s a lot of stuff to manage! Continuously!
- 7. How do we do it?
- 8. Excel
- 9. Excel Just Kidding Neo4j, of course!
- 10. So, like, seriously, how do we do it?
- 11. MacGyver
- 12. What is MacGyver? “Is that like MacGruber?”
- 13. What is MacGyver? “Is that like MacGruber?” No, but it’s on GitHub!
- 14. MacGyver provides… • What we can’t easily accomplish with Jenkins • API Integration • A batch scheduler • Data aggregation • An extensible UI • Tools for engineering, operations, infosec, QA and finance • Performs monitoring and alerting
- 15. Integrations
- 16. MacGyver + Neo4j
- 17. MacGyver + Neo4j Jenkins GitHub Load Balancer Pager Duty SAN Storage
- 18. MacGyver + Neo4j Jenkins GitHub AWS Jira Confluence HipChat Active Directory Pager Duty New RelicLoad Balancer Pager Duty SAN Storage Graphite
- 19. MacGyver + Neo4j Jenkins GitHub AWS Artifactory Jira Confluence HipChat VMWare Active Directory Pager Duty New RelicLoad Balancer Pager Duty Catchpoint SAN Storage Graphite Splunk
- 20. Why Neo4j? • Perfect for capturing/modeling interdependencies • Cypher’s ad hoc query capability can’t be beat • Easy to extend, build more relationships and layers incrementally • JSON native, FTW
- 21. How Does DevOps at Lending Club Use Neo4j? 1. App check-in/service discovery 2. Deployment and release automation 3. Infrastructure mapping
- 22. How Does DevOps at Lending Club Use Neo4j? 1. App check-in/service discovery 2. Deployment and release automation 3. Infrastructure mapping
- 23. App Check-In/Service Discovery Problem: Keeping track of many rapidly-changing services Solution: All app servers phone home to MacGyver and are stored in Neo4j as ‘App Instance’ nodes
- 26. App Check-In/Service Discovery Real time database of deployed infrastructure New services get auto-discovered by MacGyver Low maintenance Easy scalability Low latency ad hoc query capability
- 27. App Check-In/Service Discovery “Show me all the instances of app X in environment Y”
- 28. How Does DevOps at Lending Club Use Neo4j? 1. App check-in/service discovery 2. Deployment and release automation 3. Infrastructure mapping
- 29. Deployment and Release Automation Problem: Highly manual and tedious releases Difficult to answer questions like: – What pool should I deploy to? – Is the most recent revision ‘live’ right now? – Are live pool revisions in sync in different environments?
- 30. Deployment and Release Automation Solution: Utilize app check-ins and Neo4j to expose info about live and dark pools, enabling us to automate deployments in a release
- 31. Deployment and Release Automation • Blue-green deployment Server 1 Server 2 Server 3 Server 5 Server 4 Server 6 Server 7 Server 8 Service Group “Live” Pool “Dark” Pool
- 32. Deployment and Release Automation • Blue-green deployment Server 1 Server 2 Server 3 Server 5 Server 4 Server 6 Server 7 Server 8 Service Group
- 33. Deployment and Release Automation • Blue-green deployment Server 1 Server 2 Server 3 Server 5 Server 4 Server 6 Server 7 Server 8 Service Group “Draining” Pool “Live” Pool
- 34. Deployment and Release Automation • Blue-green deployment Server 1 Server 2 Server 3 Server 5 Server 4 Server 6 Server 7 Server 8 Service Group Pool Cut-over
- 35. App Instances MacGyver Neo4j <<update app instance>> MacGyver
- 36. Virtual Servers MacGyver Neo4j Load Balancer<<REST API Polling>>
- 37. Virtual Servers MacGyver Neo4j <<update virtual server>> Load Balancer<<REST API Polling>>
- 39. Virtual Service
- 40. Deployment and Release Automation Problem: Highly manual and tedious releases Difficult to answer questions like: – What pool should I deploy to? – Is the most recent revision ‘live’ right now? – Are live pool revisions in sync in different environments?
- 41. Deployment and Release Automation “What pool should I deploy to?”
- 42. Deployment and Release Automation “Are live pool revisions in sync in different environments?”
- 43. Deployment and Release Automation “Do multiple revisions exist within a single pool?”
- 44. Deployment and Release Automation “Do multiple revisions exist within a single pool?” Think about the Knight Capital meltdown in 2012!
- 45. App Instance App Instance App Instance App Instance Deployment and Release Automation MacGyver Neo4j Load Balancer <<PUT>> <<REST API>> <<REST API>>
- 46. How Does DevOps at Lending Club Use Neo4j? 1. App check-in/service discovery 2. Deployment and release automation 3. Infrastructure mapping
- 48. “Are any servers in a live pool degraded?”
- 50. “Do we have a single point of failure among any of our services?”
- 52. “If storage volume #3 goes down, what services will be impacted?”
- 53. Other Use Cases • “Are servers in the correct security zones?” • “What is the correct AWS VPC placement for this application?” • Deliver abstraction across multiple Load Balancer implementations • etc.
- 54. For More Info • https://github.com/if6was9/macgyver • https://macgyver.readme.io • https://github.com/if6was9/neorx • Ashley Sun – asun@lendingclub.com, @ashleycsun • Rob Schoening - rschoening@lendingclub.com, @rschoening
- 55. Questions
Editor's Notes
- New way of designing and structuring software applications as suites of independently deployable services