Your SlideShare is downloading. ×

Flashback: QCon San Francisco 2012

2,726

Published on

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
2,726
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
6
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Sergejus Barinovas
  • 2. Why San Francisco?Learn how others are doing at scaleLearn what problems others haveLearn does their solutions apply to usLearn does their problems apply to us
  • 3. Why San Francisco?Silicon Valley based companies: - Google - Pinterest - Facebook - Quora - Twitter - tons of others... - Netflix
  • 4. NoSQL: Past, Present, FutureEric Brewer – author of CAP theoremCP vs. AP but only on time-out (failure) ,
  • 5. Real-time Web with
  • 6. Real-time webnode.js – de-facto for real-time webopen connection for user and leave open for himweb sockets are great, but use fallbacks - mobile devices doesnt support web sockets - long polling, infinite frame, etc.more companies moving to SPDY protocol
  • 7. on mobile
  • 8. Quora on mobilefirst iPhone app - mobile app is like old app shipped on CD - hybrid application - native code for controls and navigation - HTML for viewing Q&A from the site - separate mobile optimized HTML layout of the web page
  • 9. Quora on mobilesecond Android app - created clone of iPhone app - failed! - UI natural on iPhone is alien on Android - bought Android devices and learned their philosophy - used new Google Android UI design guidelines - created new app with native for Android look & feel - users in India pay per MB, so had to optimize traffic - optimizations applied for iPhone app and web page
  • 10. Quora on mobilemobile first experience - mobile has very unique requirements - if youre good on mobile, youre good anywhere - dont use mobile app on tablets, create separate or use web
  • 11. Continuous delivery
  • 12. Continuous deliveryJesse Robbins, author of Chefinfrastructure as code - full stack automation - datacenter API (for provisioning VMs, etc.) - infrastructure is a product and app is a customer
  • 13. Continuous deliveryapplication as services - service orientation - software resiliency - deep instrumentationdev / ops as teams - service owners - shared metrics / monitoring - continuous integration / deployment
  • 14. Release engineering at
  • 15. Release engineering at FacebookChuck Rossi – release engineering managerdeployment process - teams are not deploying to production by them selves - for communication during deployment IRC is used - if team member is not connected to IRC, release is skipped - BitTorrent for deployments - powerful app monitoring and profiling (instrumentation)
  • 16. Release engineering at Facebookdeployment process - ability to release on subset of servers - very powerful feature flag mechanism by IP gender, age, … , - karma points for developers with down-vote buttonfacebook.com - continuously deployed internally - employees always access latest facebook.com - easy to report bug from the internal facebook.com
  • 17. Scaling
  • 18. Scaling Pintereseteverything in Amazon cloudbefore - had every possible ‘hot’ technology including MySQL, Cassandra, Mongo, Redis, Memcached, Membase, Elastic Search – FAIL - keep it simple, major re-architecting in late 2011
  • 19. Scaling PinteresetJanuary 2012 - Amazon EC2 + S3 + Akamai, ELB - 90 Web Engines + 50 API Engines - 66 sharded MySQL DBs + 66 slave replicas - 59 Redis - 51 Memcache - 1 Redis task queue + 25 task processors - sharded Solr - 6 engineers
  • 20. Scaling Pinteresetnow- Amazon EC2 + S3 + Akamai, Level3, EdgeCast, ELB- 180 Web Engines + 240 API Engines- 80 sharded MySQL DBs + 80 slave replicas- 110 Redis- 200 Memcache- 4 Redis task queues + 80 task processors- sharded Solr- 40 engineers
  • 21. Scaling Pinteresetschemeless DB design - no foreign keys - no joins - denormalized data (id + JSON data) - users, user_has_boards, boards, board_has_pins, pins - read slaves - heavy use of cache for speed & better consistencythinking of moving to their own DC
  • 22. Architectural patternsfor high availability at
  • 23. Architectural patterns for HAAdrian Cockcroft – director of architecture at Netflixarchitecture - everything in Amazon cloud in 3 availability zones - chaos Gorilla, latency Gorilla - service-based architecture, stateless micro-services - high attention for service resilience - handle dependent service unavailability or increased latencystarted open-sourcing to improve quality of the code
  • 24. Architectural patterns for HACassandra usage - 2 dedicated Cassandra teams - over 50 Casssandra clusters, over 500 nodes, over 30 TB of data, biggest cluster has 72 nodes - most write operations, for reads Memcache layer is used - moved to SSD in Amazon instead of spinning disks and cache - for ETL: read Cassandara backup files using Hadoop - can scale zero-to-500 instances in 8 minutes
  • 25. timelines at scale
  • 26. Timelines at scaleRaffi Krikorian – director of Twiters platform servicescore architecture - pull (timeline & search) and push (mobile, streams) use-cases - 300K QPS for timeline - on write use fan-out process to copy data for each use-case - timeline cache in Redis - when you tweet and you have 200 followers there will be 200 inserts to each follower timeline
  • 27. Timelines at scalecore architecture - Hadoop for batch compute and recommendation - code heavily instrumented (load times, latencies, etc.) - uses Cassandra, but moving off from it due to read times
  • 28. More infoSlides - http://qconsf.com/sf2012Videos - http://www.infoq.com/

×