Your SlideShare is downloading. ×
0
Disrupting the Storage Industry talk at SNIA Data Storage Innovation Conference
Disrupting the Storage Industry talk at SNIA Data Storage Innovation Conference
Disrupting the Storage Industry talk at SNIA Data Storage Innovation Conference
Disrupting the Storage Industry talk at SNIA Data Storage Innovation Conference
Disrupting the Storage Industry talk at SNIA Data Storage Innovation Conference
Disrupting the Storage Industry talk at SNIA Data Storage Innovation Conference
Disrupting the Storage Industry talk at SNIA Data Storage Innovation Conference
Disrupting the Storage Industry talk at SNIA Data Storage Innovation Conference
Disrupting the Storage Industry talk at SNIA Data Storage Innovation Conference
Disrupting the Storage Industry talk at SNIA Data Storage Innovation Conference
Disrupting the Storage Industry talk at SNIA Data Storage Innovation Conference
Disrupting the Storage Industry talk at SNIA Data Storage Innovation Conference
Disrupting the Storage Industry talk at SNIA Data Storage Innovation Conference
Disrupting the Storage Industry talk at SNIA Data Storage Innovation Conference
Disrupting the Storage Industry talk at SNIA Data Storage Innovation Conference
Disrupting the Storage Industry talk at SNIA Data Storage Innovation Conference
Disrupting the Storage Industry talk at SNIA Data Storage Innovation Conference
Disrupting the Storage Industry talk at SNIA Data Storage Innovation Conference
Disrupting the Storage Industry talk at SNIA Data Storage Innovation Conference
Disrupting the Storage Industry talk at SNIA Data Storage Innovation Conference
Disrupting the Storage Industry talk at SNIA Data Storage Innovation Conference
Disrupting the Storage Industry talk at SNIA Data Storage Innovation Conference
Disrupting the Storage Industry talk at SNIA Data Storage Innovation Conference
Disrupting the Storage Industry talk at SNIA Data Storage Innovation Conference
Disrupting the Storage Industry talk at SNIA Data Storage Innovation Conference
Disrupting the Storage Industry talk at SNIA Data Storage Innovation Conference
Disrupting the Storage Industry talk at SNIA Data Storage Innovation Conference
Disrupting the Storage Industry talk at SNIA Data Storage Innovation Conference
Disrupting the Storage Industry talk at SNIA Data Storage Innovation Conference
Disrupting the Storage Industry talk at SNIA Data Storage Innovation Conference
Disrupting the Storage Industry talk at SNIA Data Storage Innovation Conference
Disrupting the Storage Industry talk at SNIA Data Storage Innovation Conference
Disrupting the Storage Industry talk at SNIA Data Storage Innovation Conference
Disrupting the Storage Industry talk at SNIA Data Storage Innovation Conference
Disrupting the Storage Industry talk at SNIA Data Storage Innovation Conference
Disrupting the Storage Industry talk at SNIA Data Storage Innovation Conference
Disrupting the Storage Industry talk at SNIA Data Storage Innovation Conference
Disrupting the Storage Industry talk at SNIA Data Storage Innovation Conference
Disrupting the Storage Industry talk at SNIA Data Storage Innovation Conference
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Disrupting the Storage Industry talk at SNIA Data Storage Innovation Conference

1,796

Published on

Presented April 24th 2014 at Santa Clara www.dsicon.org event. Scalability goes from Ludicrous to Plaid.

Presented April 24th 2014 at Santa Clara www.dsicon.org event. Scalability goes from Ludicrous to Plaid.

Published in: Technology, Business
0 Comments
9 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,796
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
57
Comments
0
Likes
9
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide
  • Over the last few years I’ve been giving this talk to many difference audiences, while many can use it directly, there have been a lot of late adopter companies where the chasm is too big between where they are and the patterns I’m talking about. How can people make that transition, and what are the root assumptions that are getting in the way?
  • I started giving talks on what Netflix was doing in cloud in 2009. The first few times I presented the reaction was incredulous. People didn’t believe we would be attempting something so crazy.By 2010 as the migration continued the reaction shifted to accepting that we really were doing cloud, but that it wouldn’t work. We would be back building datacenters soon enough.In 2011 we had finished migrating Netflix to cloud and it was working well enough that people could see that it was an interesting thing to have done, but the consensus was that this was not relevant to anyone else in the industry. Netflix was a unique Unicorn case.The agility and speed of deployment started to win converts, and in 2012 the most common comment was that people would like to be doing what Netflix was doing in cloud, but there was no way to bridge the gap from where they were now, to where they needed to be.Netflix agressively open sourced it’s Cloud Native platform in 2012 and 2013, ending the year with 39 projects on Github. When I met with companies in 2013 it became common to find that they were already using some parts of NetflixOSS and were working on how to use more.
  • IT has moved from handling a few thousand POS terminals, to millions of web and mobile clients. From managing a few thousand employees to managing millions of customers and giving them a personalized experience. From fixed display ads in newspapers to real time bidding of online ads. This has driven a need to handle large scale and rapid change that are orders of magnitude bigger than before.
  • The size of a change reduced from an entire system, to a feature, to a check-in of part of a feature
  • That’s the quote I provided for the back of this book. It’s a novel. A horror story for geeks. Well known novel “The Goal” updated for the current age.The setting of the book is a medium sized manufacturing company struggling with the demands of IT.It opens with an IT manager getting a call that his VP and CIO have quit, WiFi is down, and payroll is corrupted, he has an immediate meeting with the CEO when he gets there. He is reluctantly put in charge of fixing things but it carries on going downhill for most of the book. In the end he is saved by DevOps of course, but you have to read it and give a copy to your CEO.But what happened to make IT such an issue?
  • Transcript

    • 1. Disrupting the Storage Industry Adrian Cockcroft @adrianco April 2014
    • 2. | Battery Ventures Industry perspective Cloud trends Cloud native storage architecture
    • 3. | Battery Ventures
    • 4. | Battery Ventures Typical reactions to my Netflix talks… “You guys are crazy! Can‟t believe it” – 2009 “What Netflix is doing won‟t work” – 2010 It only works for „Unicorns‟ like Netflix” – 2011 “We‟d like to do that but can‟t” – 2012 “We‟re on our way using Netflix OSS code” – 2013
    • 5. | Battery Ventures What I learned from my time at Netflix ● Speed wins in the marketplace ● Remove friction from product development ● High trust, low process, no hand-offs between teams ● Freedom and responsibility culture ● Don‟t do your own undifferentiated heavy lifting ● Use simple patterns automated by tooling ● Self service cloud makes impossible things instant
    • 6. | Battery Ventures
    • 7. | Battery Ventures Demands on IT Increased 1000x Compete or lose in the market!
    • 8. | Battery Ventures How fast can you innovate?
    • 9. | Battery Ventures Non-Cloud Product Development Months before you find out whether the product meets the need Hardware provisioning is undifferentiated heavy lifting – replace it with IaaS Business Need • Documents • Weeks Approval Process • Meetings • Weeks Hardware Purchase • Negotiations • Weeks Software Development • Specifications • Weeks Deployment and Testing • Reports • Weeks Customer Feedback • It sucks! • Weeks IaaS Cloud
    • 10. | Battery Ventures IaaS Based Product Development Weeks before you find out whether the product meets the need Software provisioning is undifferentiated heavy lifting – replace it with PaaS Business Need • Documents • Weeks Software Development • Specifications • Weeks Deployment and Testing • Reports • Days Customer Feedback • It sucks! • Days PaaS Cloud
    • 11. | Battery Ventures Process Hand-Off Steps for Product Development on IaaS Product Manager Development Team QA Integration Team Operations Deploy Team BI Analytics Team
    • 12. | Battery Ventures Process Hand-Off Steps for Feature Development on PaaS Product Manager Developer BI Analytics Team
    • 13. | Battery Ventures PaaS Based Product Feature Development Days before you find out whether the feature meets the need Building your own business apps is undifferentiated heavy lifting – use SaaS Business Need • Discussions • Days Software Development • Code • Days Customer Feedback • Fix this Bit! • Hours SaaS/ BPaaS Cloud
    • 14. | Battery Ventures What Happened? Rate of change increased Cost and size and risk of change reduced
    • 15. | Battery Ventures Observe Orient Decide Act Land grab opportunity Competitive Move Customer Pain Point Analysis JFDI Plan Response Share Plans Incremental Features Automatic Deploy Launch AB Test Model Hypotheses BIG DATA INNOVATION CULTURE CLOUD Measure Customers Continuous Delivery on Cloud
    • 16. | Battery Ventures Cloud Trends
    • 17. | Battery Ventures Cloud Enterprise IT Adoption By Simon Wardley http://enterpriseitadoption.com/ You Are Here
    • 18. | Battery Ventures What happened in the “price war” announcements? ● Demo: Live instance migration ● Sustained Usage Pricing Discount for over 25%/month ● New Google DNS Service ● New MS Windows Support ● Price cuts Storage 2.6 cents/GB/month Storage access 1 cent/10k ops 32% reduction in instance cost Cost for n1-standard == m3 ● Demo: Workspaces Cloud VDI ● New High Memory Instances r3 cheaper & bigger than m2 ● Updated Storage Instances i2 cheaper & bigger than hi1 ● Price cuts S3 2.75-3.0 cents/GB/month S3 access 0.4 cent/10k ops m3 cheaper&faster than m1 c3 cheaper&faster than c1 Google Cloud Amazon Web Services
    • 19. | Battery Ventures What Was Missing Last Week ● No big enterprise customers ● No reservation options ● Need more regions and zones ● Need lower inter-zone latency ● No SSD options ● No per minute billing ● Need simpler discount options ● Need more regions and zones ● No integrated PaaS strategy ● No instance migration ● Need update for m1 Google Cloud Amazon Web Services Too many architectural differences make using both interchangeably tricky
    • 20. | Battery Ventures How does IT get there?
    • 21. | Battery Ventures "This is the IT swamp draining manual for anyone who is neck deep in alligators.”
Adrian Cockcroft, Cloud Architect at Netflix New conference led by Gene Kim: Enterprise Devops – SF Oct 2014
    • 22. | Battery Ventures Continuous Deployment for Speed New book 2014 Flow Conference: Flowcon 2013 – See videos Flowcon 2014 – September…
    • 23. | Battery Ventures Speed wins in the marketplace
    • 24. | Battery Ventures Cloud Native Storage Architecture
    • 25. | Battery Ventures Cloud Native Storage Requirements ● Write data quickly ● Don‟t lose data ● Read back quickly ● Read back what I wrote… ● Distribute data globally ● With low cost ● And not run out of space. ● Make big profits for existing vendors
    • 26. | Battery Ventures Traditional vs. Cloud Native Storage Architectures Business Logic Database Master Fabric Storage Arrays Database Slave Fabric Storage Arrays Business Logic Cassandra Zone A nodes Cassandra Zone B nodes Cassandra Zone C nodes Cloud Object Store Backups
    • 27. | Battery Ventures Storage Node Options ● AWS – Create clusters of 100s of Cassandra nodes automatically From nothing exists to writing data 10min after launching (see live demo) ● Older Node Specifications AWS m2.4xlarge – 68GB RAM, 1Gbit net, 2x840GB disks, 500 iops AWS h1.4xlarge – 60GB RAM, 10Gbit net, 2TB SSD, 100000 iops AWS hs1.8xlarge – 117GB RAM, 10Gbit net, 24x2TB disk, 2.6Gbyte/s ● Current Node Specifications – Intel Ivybridge v2 AWS i2.xlarge – 30GB RAM, 10Gbit net, 800GB SSD, 45000 iops AWS i2.8xlarge – 244GB RAM, 10Gbit net, 8x800GB SSD, 365000 iops
    • 28. | Battery Ventures Write Data Quickly ● Apache Cassandra ● Write to RAM on local node ● Duplicate to remote node RAM ● Flush to local disk every 10 sec ● Huge sequential writes ● Immutable pre-sorted files ● Infrequent compaction merge ● Ack local copy, microseconds ● Ack remote copy, net latency ● Quorum write 2 out of 3 option ● Over 1M writes/s in production ● Scales linearly with nodes ● Netflix runs to 288 nodes/cluster ● Others over 1000 nodes How it works Speed and Scale
    • 29. | Battery Ventures Don‟t Lose Data - Durability ● Triple replication of data - one replica per building (Availability Zone) ● Hinted handoff for down nodes ● Copy immutable files to S3 for backup, remote regions for archive ● Replica checksum compares ● Efficient anti-entropy repair ● Support option from Datastax How it works
    • 30. | Battery Ventures Distribute Data Globally – Netflix Benchmark Test
    • 31. | Battery Ventures Do the Math on Costs ● Traditional Architecture Costs Nodes + Database licenses + SAN fabric switches + Storage arrays + Management tools + Replication tools + Backup tools + Tape backup drives + Off-site tape storage + Network switches + Power/cooling etc. ● Cloud Native Architecture Costs Nodes (includes storage) + Cassandra support + S3 backup ● Cloud Native Benchmarking Pay only for the hours that the cluster is running ● Cloud Native Capacity Planning Start small, grow the cluster only when it‟s filling up, no downtime
    • 32. | Battery Ventures Cloud Native Storage Requirements Met ● Write data quickly – millions of low latency ops/s ● Don‟t lose data – triple AZ replication, backups to S3 ● Read back quickly – millions of low latency ops/s ● Read back what I wrote… - checksums and automatic repairs ● Distribute globally – Cassandra cross-regional replication ● With low cost – using lots of cheap internal SSD ● And not run out of space – scales up to “plaid” (beyond ludicrous) ● Make big profits for existing vendors – sorry…
    • 33. | Battery Ventures Cloud Native for High Availability at Scale NetflixOSS at netflix.github.com and techblog.netflix.com Over 40 projects, PaaS, NoSQL, Big Data, etc.
    • 34. | Battery Ventures Priam – Cassandra co-process ● Runs alongside Cassandra on each instance ● Fully distributed, no central master coordination ● S3 Based backup and recovery automation ● Bootstrapping and automated token assignment. ● Centralized configuration management ● RESTful monitoring and metrics ● Automated online re-size to double node count
    • 35. | Battery Ventures Cassandra Astyanax Java Client Recipes ● Distributed row lock (without needing zookeeper) ● Multi-region row lock ● Uniqueness constraint ● Multi-row uniqueness constraint ● Chunked and multi-threaded large file storage ● Reverse index search ● All rows query ● Durable message queue ● Contributed: High cardinality reverse index
    • 36. | Battery Ventures Staash - Generic Data Access Layer Microservice ● Storage Tier As A Service over Http “STaaSH” ● Polyglot persistence via unified REST API ● Cassandra/Astyanax Recipe Implementations ● Cassandra and MySQL supported now ● More datastores under development… ● Will allow polyglot “join” across datastores
    • 37. | Battery Ventures What‟s Next? ● Cassandra with local disk replaces Oracle/MySQL, SAN, Array etc. ● AWS S3 and Google Data Store replace tape for backup/archive ● Cloud prices halve every two years ● Epic Google and AWS price war (spoiler: everyone else dies…) ● How and when will Google compete with AWS SSD options? ● SSD moves to the memory channel, for even lower latency
    • 38. | Battery Ventures Disclosure: Diablo Technologies is a Battery Ventures Portfolio Company See www.battery.com for a list of portfolio investments
    • 39. | Battery Ventures Any Questions? Presentations by @adrianco ● Battery Ventures http://www.battery.com ● Adrian‟s Blog http://perfcap.blogspot.com ● Netflix Tech Blog http://techblog.netflix.com ● Netflix Slideshare http://slideshare.com/netflix ● Monitorama Opening Keynote Portland OR - May 7th, 2014 ● GOTO Chicago Opening Keynote May 20th, 2014 ● DevOps Summit at Cloud Expo New York – June 10th, 2014 ● Qcon New York – June 11th, 2014 ● GOTO Copenhagen/Aarhus – Denmark – Oct 25th, 2014

    ×