Teaching IT one trick or two

1,343 views
1,267 views

Published on

New operations and technology concepts for IT

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,343
On SlideShare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
0
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Teaching IT one trick or two

  1. 1. Teaching IT New TricksTell me and I’ll forgetShow me and I may remember STKI Summit 2012Involve me and I’ll understand Pini Cohen VP and Senior Analyst
  2. 2. Prof. Clayton Christensen: Disruptive Innovation ModelRemember Digital Equipment Corporation (DEC). “Underdogs become mainstreamfaster than we think”. Change towards what looks as “non-mature” areas is crucial T1 T2 Pini Cohen’s work Copyright STKI@2012 2 Do not remove source or attribution from any slide or graph
  3. 3. New tricks - agenda • Web architecture and “philosophy” examples • Devops: CultureProcesses • Technology listing Source: http://www.b2binbound.com/blog/?Tag=paradigm%20shift Pini Cohen’s work Copyright STKI@2012 Do not remove source or attribution from any slide or graph 3
  4. 4. Pinterest Pini Cohen’s work Copyright STKI@2012 Do not remove source or attribution from any slide or graph 4
  5. 5. Pinterest Architecture Update - 18 Million Visitors, 10x Growth,12 Employees, 410TB of Data • 80 million objects stored in S3 with 410 terabytes of user data, 10x what they had in August. EC2 instances have grown by 3x. Around $39K fo S3 and $30K for EC2 a month. • Pay for what you use saves money. Most traffic happens in the afternoons and evenings, so they reduce the number of instances at night by 40%. • 12 employees as of last December. Using the cloud a site can grow dramatically while maintaining a very small team. Looks like 31 employees as of now. Source: http://highscalability.com/blog/2012/5/21/pinterest-architecture-update-18-million-visitors-10x-growth.html Pini Cohen’s work Copyright STKI@2012 Do not remove source or attribution from any slide or graph 5
  6. 6. Pinterest Architecture Update - 18 Million Visitors, 10x Growth,12 Employees, 410TB of Data • Different type of technology: • Written in Python and Django • Sharding is used, a database is split when it reaches 50% of capacity, allows easy growth and gives sufficient IO capacity • ELB (Elastic Load Balancing) is used to load balance across instances. The ELB API makes it easy to move instances in and out of production. • Hadoop-based Elastic Map Reduce is used for data analysis and costs only a few hundred dollars a month. Source: http://highscalability.com/blog/2012/5/21/pinterest-architecture-update-18-million-visitors-10x-growth.html Pini Cohen’s work Copyright STKI@2012 Do not remove source or attribution from any slide or graph 6
  7. 7. Instagram • The Instagram philosophy: • Simplicity • Optimized for minimal operational burden • Instrument everything Pini Cohen’s work Copyright STKI@2012 Do not remove source or attribution from any slide or graph 7
  8. 8. Scaling Instagram • Instagram went to 30+ million users in less than two years and then rocketed to 40 million users 10 days after the launch of its Android application. • After the release of the Android they had 1 million new users in 12 hours. • 2 engineers in 2010. • 3 engineers in 2011 • 5 engineers 2012, 2.5 on the backend. This includes iPhone and Android development. Source: http://highscalability.com/blog/2012/4/16/instagram-architecture-update-whats-new-with-instagram.html Pini Cohen’s work Copyright STKI@2012 Do not remove source or attribution from any slide or graph 8
  9. 9. Instagram technology philosophy • Engineer solutions you are not constantly returning to because they are broke. STKI: operations operations operations! • Try not to have two tools doing the same job. • Focus on making what you have better: fast, beautiful, photo sharing. Make the fast part faster. Can we all of our requests take 50% of the time? • Only add software to your stack that is optimized for operational simplicity • Ideas are disposable: if one doesn’t work, you quickly move on to another. • Have fun Source:http://highscalability.com/blog/2012/4/16/instagram-architecture-update-whats-new-with-instagram.html stki modifications Pini Cohen’s work Copyright STKI@2012 Do not remove source or attribution from any slide or graph 9
  10. 10. Instagram is down! Pini Cohen’s work Copyright STKI@2012 Do not remove source or attribution from any slide or graph 10
  11. 11. Tumblr – Microbloging social networking platform • 500 million page views a day • 15B+ page views month • Peak rate of ~40k requests per second • 1+ TB/day into Hadoop cluster • Many TB/day into MySQL/HBase/Redis/Memcache • Growing at 30% a month • ~1000 hardware nodes in production (not cloud) • ~20 engineers (total 106 employees) Source: http://highscalability.com/blog/2012/2/13/tumblr-architecture-15-billion-page-views-a-month-and-harder.html STKI modifications Pini Cohen’s work Copyright STKI@2012 Do not remove source or attribution from any slide or graph 11
  12. 12. New tricks - agenda • Web architecture examples • Devops: CultureProcesses • Technology listing Source: http://www.b2binbound.com/blog/?Tag=paradigm%20shift Pini Cohen’s work Copyright STKI@2012 Do not remove source or attribution from any slide or graph 12
  13. 13. Devops – Development and Operations • Addresses the conflict between Development and Operations: • Development – are paid for change • Operations – change is the enemy! • “Wall of Confusion” - combination of conflicting motivations, processes, and tooling Source: http://dev2ops.org/blog/2010/2/22/what-is-devops.html Pini Cohen’s work Copyright STKI@2012 Do not remove source or attribution from any slide or graph 13
  14. 14. Devops aims at: Source: http://dev2ops.org/blog/2010/2/22/what-is-devops.html • DevOps enables the benefits of Agile development to be felt at the organizational level. DevOps does this by allowing for fast and responsive, yet stable, operations that can be kept in sync with the pace of innovation coming out of the development process. http://en.wikipedia.org/wiki/File:Devops.png Pini Cohen’s work Copyright STKI@2012 Do not remove source or attribution from any slide or graph 14
  15. 15. Striving towards Devops state of mind: • Measurement and incentives to change culture - metrics based on joint performance • Unified processes • Unified tooling • Rolling back should be easy Pini Cohen’s work Copyright STKI@2012 Do not remove source or attribution from any slide or graph 15
  16. 16. Devops needs automation. Example: Opscode - Chef • With Chef, you write abstract definitions as source code to describe how you want each part of your infrastructure to be built, and then apply those descriptions to individual servers. • The result is a fully automated infrastructure: when a new server comes on line, the only thing you have to do is tell Chef what role it should play in your architecture. Source: opscode Pini Cohen’s work Copyright STKI@2012 Do not remove source or attribution from any slide or graph 16
  17. 17. Continuous deployment: The Lean Startup Philosophy • The Lean Startup relies on validated learning, scientific experimentation, and iterative product releases to shorten product development cycles, measure progress, and gain valuable customer feedback. (from wikipedia STKI modifications) • The basic insight : “I wrote small feature but the customer thinks it is BIG” (and vice versa). Continuous Deployment: taking Devops to the extreme! Delivery of new functionality is core value for new companies Pini Cohen’s work Copyright STKI@2012 Do not remove source or attribution from any slide or graph 17
  18. 18. Continues deployment: your code is always stable! • Old school deployment Compile Code is RunStable not stable Code is Development Pass Tests Deploy code for 2-3 stable (manuala weeks utomatic) • Continues deployment Compile Run Deploy If errorStable Development automatical accrues Code is Pass tests – all code “2 lines” ly several change stable automatically times a day immediately • What about the “buffer overflow” after 5 days?! • Answer: you will have it anyway! Pini Cohen’s work Copyright STKI@2012 Do not remove source or attribution from any slide or graph 18
  19. 19. Continues integrationdeployment perquisites • Tools needed • Version control server (SCM) • Build server • Deployment Server • Automation tools • Continouse integration tools • Work Process • Commit/Update often (each change) • Run often • Tests • Run test often • Common code ownership Source: http://www.google.co.il/url?sa=t&rct=j&q=Continuous+Integration+%28CI%29+with+Hudson++filetype%3Appt&source=web&cd=1&ved=0CEoQFjAA&url=http%3A%2F%2Fpretoria.uoc.es%2Fwpmu%2FblogaTE%2Ffile s%2F2010%2F06%2Fcontinuous-integration-with-hudson1.ppt&ei=UHH5T7aaJ-jQ4QTVhcWNBw&usg=AFQjCNHpiw711JVy9crKXXnJ0hwnw95E1g STKI modifications Pini Cohen’s work Copyright STKI@2012 Do not remove source or attribution from any slide or graph 19
  20. 20. Devops vs. Private Cloud? • In many aspects the objectives of Devops and Private Cloud are overlapping • Automation is at the core of both Private Cloud and Devops • Standardization is a must for automation Source: http://www.pistoncloud.com/2012/01/devops-and-private-cloud-sitting-in-a-tree/ Pini Cohen’s work Copyright STKI@2012 Do not remove source or attribution from any slide or graph 20
  21. 21. Old vs. NewOld Datacenter New CloudDatacenterLicensed and Installed Applications SaaS (Workday, Pagerduty (Central SQL Database Distributed Key/Value NoSQLSticky In-Memory Session Shared Memory Cache Session Source: http://highscalability.com/blog/2012/5/7/startups-are-creating-a-new-system-of-the-world-for-it.html stki modificationsTangled Service Interfaces Layered Service InterfacesInstrumented Code Instrumented Service PatternsFat Complex Objects Lightweight Serialized ObjectsChatty Protocols Latency Tolerant ProtocolsManual and Static Tools Automated and Scalable ToolsMonolithic Software Development Teams Organized around ServicesMonolithic Applications Building Your Own PaaSStatic and Slow Growing Capacity Incremental and Fast Growing CapacityHeavy Process/Meetings/Tickets/Waiting Better Business AgilitySingle Location Massive Geographical DistributionFocus on How Much it Costs Focus on How Much Value it BringsOwnership/CapEx Leasing/OpEx/Spot/Reserved/On Demand Pini Cohen’s work Copyright STKI@2012 Do not remove source or attribution from any slide or graph 21
  22. 22. Technology listing • Hadoop Mapreduce • NoSQL dbms (Cassandra, Mongo, HBASE) • Shrading • In Memory DBMS • DBMSDW appliances • Memcashed • MemSQL • Solr • Redis • DJANGO • ELB - Elastic load balancing amazon Pini Cohen’s work Copyright STKI@2012 Do not remove source or attribution from any slide or graph
  23. 23. Summary• Open your mind – things are happening out there!• “Underdogs become mainstream faster than we think”.• Change towards what looks as “non- mature” areas is crucial• Standardization and Automation is a must• Categorization of Infra-Ops activities: • Faults • Specific work (on “Neptun” for APP X) Source: http://toktobis.deviantart.com/art/Open-Your-Mind-109225982 • General work (all Web Servers) Pini Cohen’s work Copyright STKI@2012 Do not remove source or attribution from any slide or graph 23
  24. 24. Thanks for your patience and hope you enjoyed Pini Cohen’s work Copyright STKI@2012 24 Do not remove source or attribution from any slide or graph

×