Cloud Data Integration


Published on

Rob May's presentation at Cloud Expo about Cloud Data Integration.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Could break this into: (1) it was hard and (2) it was easy
  • Mention that you didn’t fail when AWS did. Talk about why.
  • Cloud Data Integration

    1. 1.<br />Board Meeting December 2010<br />Rob May (@robmay)<br />CEO, Backupify <br /><br /> <br />Confidential<br />Agile Cloud Integration<br />1<br />
    2. 2. What You Will Learn<br />How to integrate at the speed of business<br />How to avoid data chaos<br />Best practices for cloud integration<br />2<br />
    3. 3. About Backupify<br />Backup, Search, Restore SaaS data<br />Founded in 2008<br />165,000 customers<br />150TB of data<br />Integrated with 18 services<br />One of the largest users of Cassandra<br />3<br />
    4. 4. CONFIDENTIAL<br />4<br />30%<br />Data Loss is Due to User Error<br />
    5. 5. 5<br />“<br />Yes, I clicked the delete button. Yes, I regret it now. Yes, I belatedly realized why the Backupify service makes sense.<br />”<br />Backup For Cloud AppsDavid F. Carr<br />November 2010<br />
    6. 6. What Do We Mean By Integration?<br />6<br />
    7. 7. Back In The Day (On Premise)<br />Few applications had good APIs but…<br />Integration was relatively easy<br />Full control of access, security & storage<br />Independent access to all data in DBMSs & FS<br />DBMS & FS triggers could kick off integration<br />7<br />
    8. 8. Back In The Day (On Premise)<br />Pros<br />Many good integration products with mature application connectors<br />Ability to govern software updates across systems<br />You had more control<br />Could always go straight at the data<br />Cons<br />Too many plumbing options (CORBA, DCOM, RMI, SOAP, MQ, …)<br />Few apps had good APIs<br />Version management problems<br />8<br />
    9. 9. Moving To SaaS Means<br />You have all the APIs you’ll ever need<br />Everything is really simple<br />9<br />
    10. 10. Benefits of Moving To SaaS<br />Lower costs<br />Information accessible anywhere<br />Productive users<br />Fewer maintenance headaches<br />10<br />
    11. 11. Cons of Moving To SaaS<br />Loss of access<br />Service outages & data loss<br />Data is isolated<br />The API is what it is<br />Loss of control<br />Forced app/API upgrades<br />Hacked accounts<br />Privacy risk<br />11<br />
    12. 12. The Problem With APIs<br />Design issues<br />Vendors design for the expected use cases<br />Customers always push the boundaries<br />Hard to build consistent copy of data model<br />Implementation may not follow spec<br />Access limitations<br />API Throttling<br />Query result set size limitations<br />Historical query limitations<br />12<br />
    13. 13. The Solution: Cloud Data Integration<br />The best of both worlds<br />Use SaaS as much as you want<br />Use SaaS APIs whenever you can<br />Have full control & independent access to your SaaS & on-premise data in the cloud<br />Rules, triggers & integration logic run right next to the data in the cloud<br />13<br />
    14. 14. The Problem With Cloud Integration<br />Cloud integration == Connectors 2.0<br />14<br />
    15. 15. This is Really Hard<br />Massive infrastructure challenges<br />Backupify has accidently become one of the foremost experts on Cassandra<br />It is easy to screw up<br />E.g., lock users out of their Gmail accounts<br />Some things are impossible<br />How do you Twitter in regulated industries?<br />15<br />
    16. 16. The Hard Problems are Worth Solving<br />Cloud Data Integration is agile integration<br />Best of both worlds<br />Best way to respond to new business needs<br />Best way to protect against vendor change<br />The SaaS vendors will help<br />Front vs. backdoor SaaS APIs<br />Cloud Data Intermediaries, e.g., GNIP<br />16<br />
    17. 17. How To Think About Cloud Data Integration<br />17<br />
    18. 18. The Hard Skills<br />API knowledge & know-how<br />Extreme & continuous testing automation<br />Efficient cloud operations at huge scale<br />Design for failure<br />Avoid data chaos<br />18<br />
    19. 19. The Soft Skills: Be A Good Partner<br />19<br />
    20. 20. Best Practices<br />Implement, then generify<br />Play nice with partners<br />Expect to throttle<br />Authenticate securely<br />Plan for data model/access changes<br />CONFIDENTIAL<br />20<br />
    21. 21. Moving At The Speed of Business<br />Generify you architecture as you incorporate more APIs<br />Build for fault tolerance<br />Use Oauth or similar authentication options when possible<br />CONFIDENTIAL<br />21<br />
    22. 22. Avoid Data Chaos<br />Build throttling into your model – you will eventually need it.<br />Plan for a time when the API data model will allow more or less access to data<br />Play nice with partners, but be vocal – they don’t always get it right.<br />CONFIDENTIAL<br />22<br />
    23. 23. What Else Can You Do with Your Data?<br />23<br />Backup<br />Search<br />Migrate<br />Restore<br />Integrate<br />Cloud Data Integration<br />
    24. 24. Questions?<br />Contact me:<br />@robmay<br /><br /> <br />CONFIDENTIAL<br />24<br />