Enterprise IT: What's Beyond Virtualization
•
6 likes•639 views
Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/1o0X6ov. Derek Collison discusses some of the technologies and approaches for building a self-healing infrastructure: Intelligent layer 7 SDN with semantic awareness, distributed scheduling algorithms, policy distribution and invalidation, health monitoring and management, self healing techniques, and the role of unsupervised deep machine learning and anomaly detection. Filmed at qconnewyork.com. An industry veteran and pioneer in large-scale distributed systems and enterprise computing, Derek Collison has held executive positions at TIBCO Software, Google, and VMware. He co-founded the AJAX APIs group at Google and went on to VMware to design and architect the industry's first open PaaS, Cloud Foundry.
Recommended
More Related Content
Viewers also liked
More from C4Media
More from C4Media (20)
Recently uploaded
Recently uploaded (20)
Enterprise IT: What's Beyond Virtualization
- 1. Beyond Virtualization Derek Collison - Apcera, Inc.! @derekcollison! ! June 12, 2014 - QCon New York
- 2. Watch the video with slide synchronization on InfoQ.com! http://www.infoq.com/presentations /beyond-virtualization InfoQ.com: News & Community Site • 750,000 unique visitors/month • Published in 4 languages (English, Chinese, Japanese and Brazilian Portuguese) • Post content from our QCon conferences • News 15-20 / week • Articles 3-4 / week • Presentations (videos) 12-15 / week • Interviews 2-3 / week • Books 1 / month
- 3. Presented at QCon New York www.qconnewyork.com Purpose of QCon - to empower software development by facilitating the spread of knowledge and innovation Strategy - practitioner-driven conference designed for YOU: influencers of change and innovation in your teams - speakers and topics driving the evolution and innovation - connecting and catalyzing the influencers and innovators Highlights - attended by more than 12,000 delegates since 2007 - held in 9 cities worldwide
- 4. About ! ! • Architected and built TIBCO Rendezvous and EMS Messaging Systems! • Co-founded AJAX APIs group at Google! • Designed and built Cloud Foundry! • Founder and CEO at Apcera! • Inspiration: Fast Distributed Systems @derekcollison 2 QCon NY: “Beyond Virtualization” Derek Collison
- 5. The future of enterprise IT lies beyond virtualization @derekcollison 3 QCon NY: “Beyond Virtualization”
- 6. Virtualization == @derekcollison 4 QCon NY: “Beyond Virtualization”
- 7. EVERYTHING is a distributed system these days @derekcollison 5 QCon NY: “Beyond Virtualization”
- 8. So orchestration and composing systems will define the future @derekcollison 6 QCon NY: “Beyond Virtualization”
- 9. To look into the future Let’s see where we are @derekcollison 7 QCon NY: “Beyond Virtualization”
- 10. IT Today Old school Virtualization IaaS IaaS, SaaS, PaaS Cloud @derekcollison 8 QCon NY: “Beyond Virtualization” ?
- 11. We care about what’s next @derekcollison 9 QCon NY: “Beyond Virtualization” ?
- 12. Automate undifferentiated heavy lifting, speed up the mundane @derekcollison 10 QCon NY: “Beyond Virtualization”
- 13. Orchestrate Secure and Compliant Composeable Systems @derekcollison 11 QCon NY: “Beyond Virtualization”
- 14. Align the value to you with the value to your organization @derekcollison 12 QCon NY: “Beyond Virtualization”
- 15. Build what you need.. @derekcollison 13 QCon NY: “Beyond Virtualization”
- 16. Assemble the rest @derekcollison 14 QCon NY: “Beyond Virtualization”
- 17. PaaS helps @derekcollison 15 QCon NY: “Beyond Virtualization”
- 18. PaaS Helps @derekcollison 16 QCon NY: “Beyond Virtualization” • Tries to speed up deployment! • Preset, biased approach! • Only a small piece of the puzzle! - Enterprises need lifecycle management, security, compliance, governance, etc.
- 19. PaaS is Not Enough http://apcera.com/blog/paas-is-not-enough/ @derekcollison 17 QCon NY: “Beyond Virtualization”
- 20. Docker helps @derekcollison 18 QCon NY: “Beyond Virtualization”
- 21. Docker Helps @derekcollison 19 QCon NY: “Beyond Virtualization” • The dawn of the composeable enterprise! • More control over the pieces! • Great Ecosystem!
- 22. DockerCon Initiatives @derekcollison 20 QCon NY: “Beyond Virtualization” • libSwarm! • libContainer! • libChan
- 23. Docker The Future @derekcollison 21 QCon NY: “Beyond Virtualization” • Identity! • Authorization! • Trust
- 24. Docker TBDs @derekcollison 22 QCon NY: “Beyond Virtualization” • How to compose and orchestrate the system?! • etcd? confd?! • Make it transparent! • Don’t make me rewrite! • libSwarm, libChan?! • What about compliance?! • Heartbleed?! • Linux zero-day exploit?! • Tell me if I am compliant! • Tell me what is at risk
- 25. We Want Things to Just Work @derekcollison 23 QCon NY: “Beyond Virtualization” • Self Service! • Composeable Systems (legos)! • Faster Iterative Development! • Faster Deployments! • Fault Tolerance! • High Availability! • Guaranteed SLAs
- 26. We’re getting there @derekcollison 24 QCon NY: “Beyond Virtualization”
- 27. The Future of IT @derekcollison 25 QCon NY: “Beyond Virtualization” • Declarative! • Composeable! • Extreme Agility! • Security and Compliance - Transparently! • Fluid and Abstracted Infrastructure and Services! • Multiple delivery models in one system
- 28. Declarative talks to talks to B CC @derekcollison 26 QCon NY: “Beyond Virtualization” • App A needs: ! - X memory and Y CPU! - N storage! - I/O SLAs for talking to B and C! - available URL for trusted identities! - run on premise, co-located near B App A B C
- 29. Intelligent workloads @derekcollison 27 QCon NY: “Beyond Virtualization” App A
- 30. Intelligent systems @derekcollison 28 QCon NY: “Beyond Virtualization” App A
- 31. Where do we start? @derekcollison 29 QCon NY: “Beyond Virtualization”
- 32. Required Functionality @derekcollison 30 QCon NY: “Beyond Virtualization” • What App A needs! ! • Where App A runs! ! • How App A finds B and C! ! • How others find App A! ! • What happens on failures
- 33. Required Functionality @derekcollison 31 QCon NY: “Beyond Virtualization” • What App A needs Packaging & Dependencies! • Where App A runs Provisioning & Scheduling! • How App A finds B and C Addressing & Discovery! • How others find App A External Mapping! • What happens on failures Monitoring & Management
- 34. Packaging & Dependencies @derekcollison 32 QCon NY: “Beyond Virtualization” • What the job needs to run! • Changes from Dev to Prod! • Runtimes, OS, libraries! • Who defines what these are! • Whether existing tools are sufficient for consistency, compliance, auditing! - SCCS and Chef / Puppet! - AMIs or VMDKs! - Docker Images App A DEV PROD runtimes! OS! libraries runtimes! OS! libraries
- 35. Provisioning & Scheduling @derekcollison 33 QCon NY: “Beyond Virtualization” • Where workloads run! • Network perimeter security models! • Unit of work: VM, App, Image! • Automatic, instantaneous and transparent policy compliance! • Compliance and deployment handled independently! • New tools: Mesos, Fleet, Diego 10 weeks 2 min. 500ms human! behavior ! change Speed
- 36. Addressing & Discovery ✓ ✓ @derekcollison 34 QCon NY: “Beyond Virtualization” • DNS is insufficient - inside! • Needs to fit what we have, without changing apps! • System reacts as things move! • Load balancing! • Scaling up and down ETCD / CONFD External Internal X ✓ ✓ Router Router
- 37. External Mapping • HTTP/TCP connectivity! • How do you find something? ! • Load balancing! • Rapid scaling! • Health monitoring and repair! • DNS sufficient for external, but not internal External Internal ✓ ✓ @derekcollison 35 QCon NY: “Beyond Virtualization” X ✓ ✓ Router Router
- 38. Monitoring & Management @derekcollison 36 QCon NY: “Beyond Virtualization” • What happens when something fails?! • Manual or Automatic?! • Who determines failure and whether we trust the system! • Its sick, not dead! - Latency vs. Chaos monkey! • Measure the effect of change beforehand?! • Extensible & Pluggable BORG / Omega Chaos Latency
- 39. Bolt-on is not the way to get there @derekcollison 37 QCon NY: “Beyond Virtualization”
- 40. What we need is a platform OS @derekcollison 38 QCon NY: “Beyond Virtualization”
- 41. Programmable, pluggable, and composeable from the inside out @derekcollison 39 QCon NY: “Beyond Virtualization”
- 42. The secure, hybrid, trusted platform OS for multi-datacenter @derekcollison 40 QCon NY: “Beyond Virtualization”
- 43. A Platform OS @derekcollison 41 QCon NY: “Beyond Virtualization” • All resources in a common pool! • Real-time networking, addressing, and discovery! • Awareness of ontologies AND communication semantics! • Contextual security and policy just work! • Built for rapid change - all change! • Policy-compliant resource isolation, connectivity, and SLAs CC App A C talks to pattern data behavior policy! on the fly
- 44. We Have the Right Pieces @derekcollison 42 QCon NY: “Beyond Virtualization” • Isolation Contexts - Docker! • SDN - Software-Defined Networking! • Management and Resource Pooling (CMPs)! • Intelligent and Compliant Job Scheduling! • Intelligent Canarying, A/B rollouts and testing Just not in one place
- 45. Isolation Context • Isolation Context: isolated, insulated, autonomous! • Speed and weight! - Hypervisors for virtualization! - LXC, libContainer (containers) - Docker! - Micro-task virtualization! • Google chargeback diversion Faster, more lightweight and purpose-built Virtualization Containerization Micro-task Virtualization @derekcollison 43 QCon NY: “Beyond Virtualization”
- 46. SDN - Software-Defined Networking @derekcollison 44 QCon NY: “Beyond Virtualization” • Network perimeter security! • Application-level changes! • Layer 7 semantics! - How many INSERTS per second from all of App A?! - Can I disallow DROP and DELETE calls between 1-3AM?! • Compliant and transparent network! - It just works, e.g. mobile
- 47. Intelligent, Compliant Job Scheduling • Pick the best place to run for a given job and policy! • How the system rebalances and utilizes new resources! • Centralized or Distributed algorithms! • How policy affects decision-making (e.g., geography)! • New tools: Mesos, Fleet, Diego @derekcollison 45 QCon NY: “Beyond Virtualization”
- 48. Intelligent Canarying Prod • Measured rollout success! • A/B testing! • Blue-green deployments ! • Automated rollout and rollback App A v2 10% traffic Rollout Rollback @derekcollison 46 QCon NY: “Beyond Virtualization” Dev 90% traffic App A v1
- 49. Intelligent Canarying • A lot of data needed! - resource utilizations: CPU, Mem, Storage! - communication patterns: cascading effects! - temporal awareness! • All data will feed into automated, anomaly detection services! - Utilizing unsupervised deep machine learning @derekcollison 47 QCon NY: “Beyond Virtualization”
- 50. The Future of IT - Platform OS Diverse Workloads! (e.g., apps, services) Provisioning! Scheduling! Health Monitoring! Addressing! Discovery Governance! Compliance! Security! Automation! Orchestration IaaS @derekcollison 48 QCon NY: “Beyond Virtualization” Hardware Internal Services External Services One Platform
- 51. Summary @derekcollison 49 QCon NY: “Beyond Virtualization”
- 52. Summary @derekcollison 50 QCon NY: “Beyond Virtualization” • Composeable platforms! • Intelligent workloads sans code changes! • Policy aware...! - Packaging and Dependency Management ! - Job Scheduling and Provisioning! - Addressing, Discovery, Networking! - Monitoring and Management! - Lifecycle Management and Intelligent Canarying A POLICY OF INNOVATION
- 53. Resources @derekcollison 51 QCon NY: “Beyond Virtualization” • Docker - https://www.docker.io! • Mesos - http://mesos.apache.org! • CoreOS - https://coreos.com! • Fleet, Etcd - https://coreos.com/using-coreos/etcd! • Consul - http://www.consul.io! • Continuum - http://apcera.com/continuum
- 54. Thank You Derek Collison - Apcera, Inc.! @derekcollison! ! June 12, 2014 - QCon New York
- 55. Watch the video with slide synchronization on InfoQ.com! http://www.infoq.com/presentations/beyond-virtualization