Your SlideShare is downloading. ×
0
Flashback: QCon San Francisco 2012
Flashback: QCon San Francisco 2012
Flashback: QCon San Francisco 2012
Flashback: QCon San Francisco 2012
Flashback: QCon San Francisco 2012
Flashback: QCon San Francisco 2012
Flashback: QCon San Francisco 2012
Flashback: QCon San Francisco 2012
Flashback: QCon San Francisco 2012
Flashback: QCon San Francisco 2012
Flashback: QCon San Francisco 2012
Flashback: QCon San Francisco 2012
Flashback: QCon San Francisco 2012
Flashback: QCon San Francisco 2012
Flashback: QCon San Francisco 2012
Flashback: QCon San Francisco 2012
Flashback: QCon San Francisco 2012
Flashback: QCon San Francisco 2012
Flashback: QCon San Francisco 2012
Flashback: QCon San Francisco 2012
Flashback: QCon San Francisco 2012
Flashback: QCon San Francisco 2012
Flashback: QCon San Francisco 2012
Flashback: QCon San Francisco 2012
Flashback: QCon San Francisco 2012
Flashback: QCon San Francisco 2012
Flashback: QCon San Francisco 2012
Flashback: QCon San Francisco 2012
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Flashback: QCon San Francisco 2012

2,737

Published on

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
2,737
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
6
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  1. Sergejus Barinovas
  2. Why San Francisco?Learn how others are doing at scaleLearn what problems others haveLearn does their solutions apply to usLearn does their problems apply to us
  3. Why San Francisco?Silicon Valley based companies: - Google - Pinterest - Facebook - Quora - Twitter - tons of others... - Netflix
  4. NoSQL: Past, Present, FutureEric Brewer – author of CAP theoremCP vs. AP but only on time-out (failure) ,
  5. Real-time Web with
  6. Real-time webnode.js – de-facto for real-time webopen connection for user and leave open for himweb sockets are great, but use fallbacks - mobile devices doesnt support web sockets - long polling, infinite frame, etc.more companies moving to SPDY protocol
  7. on mobile
  8. Quora on mobilefirst iPhone app - mobile app is like old app shipped on CD - hybrid application - native code for controls and navigation - HTML for viewing Q&A from the site - separate mobile optimized HTML layout of the web page
  9. Quora on mobilesecond Android app - created clone of iPhone app - failed! - UI natural on iPhone is alien on Android - bought Android devices and learned their philosophy - used new Google Android UI design guidelines - created new app with native for Android look & feel - users in India pay per MB, so had to optimize traffic - optimizations applied for iPhone app and web page
  10. Quora on mobilemobile first experience - mobile has very unique requirements - if youre good on mobile, youre good anywhere - dont use mobile app on tablets, create separate or use web
  11. Continuous delivery
  12. Continuous deliveryJesse Robbins, author of Chefinfrastructure as code - full stack automation - datacenter API (for provisioning VMs, etc.) - infrastructure is a product and app is a customer
  13. Continuous deliveryapplication as services - service orientation - software resiliency - deep instrumentationdev / ops as teams - service owners - shared metrics / monitoring - continuous integration / deployment
  14. Release engineering at
  15. Release engineering at FacebookChuck Rossi – release engineering managerdeployment process - teams are not deploying to production by them selves - for communication during deployment IRC is used - if team member is not connected to IRC, release is skipped - BitTorrent for deployments - powerful app monitoring and profiling (instrumentation)
  16. Release engineering at Facebookdeployment process - ability to release on subset of servers - very powerful feature flag mechanism by IP gender, age, … , - karma points for developers with down-vote buttonfacebook.com - continuously deployed internally - employees always access latest facebook.com - easy to report bug from the internal facebook.com
  17. Scaling
  18. Scaling Pintereseteverything in Amazon cloudbefore - had every possible ‘hot’ technology including MySQL, Cassandra, Mongo, Redis, Memcached, Membase, Elastic Search – FAIL - keep it simple, major re-architecting in late 2011
  19. Scaling PinteresetJanuary 2012 - Amazon EC2 + S3 + Akamai, ELB - 90 Web Engines + 50 API Engines - 66 sharded MySQL DBs + 66 slave replicas - 59 Redis - 51 Memcache - 1 Redis task queue + 25 task processors - sharded Solr - 6 engineers
  20. Scaling Pinteresetnow- Amazon EC2 + S3 + Akamai, Level3, EdgeCast, ELB- 180 Web Engines + 240 API Engines- 80 sharded MySQL DBs + 80 slave replicas- 110 Redis- 200 Memcache- 4 Redis task queues + 80 task processors- sharded Solr- 40 engineers
  21. Scaling Pinteresetschemeless DB design - no foreign keys - no joins - denormalized data (id + JSON data) - users, user_has_boards, boards, board_has_pins, pins - read slaves - heavy use of cache for speed & better consistencythinking of moving to their own DC
  22. Architectural patternsfor high availability at
  23. Architectural patterns for HAAdrian Cockcroft – director of architecture at Netflixarchitecture - everything in Amazon cloud in 3 availability zones - chaos Gorilla, latency Gorilla - service-based architecture, stateless micro-services - high attention for service resilience - handle dependent service unavailability or increased latencystarted open-sourcing to improve quality of the code
  24. Architectural patterns for HACassandra usage - 2 dedicated Cassandra teams - over 50 Casssandra clusters, over 500 nodes, over 30 TB of data, biggest cluster has 72 nodes - most write operations, for reads Memcache layer is used - moved to SSD in Amazon instead of spinning disks and cache - for ETL: read Cassandara backup files using Hadoop - can scale zero-to-500 instances in 8 minutes
  25. timelines at scale
  26. Timelines at scaleRaffi Krikorian – director of Twiters platform servicescore architecture - pull (timeline & search) and push (mobile, streams) use-cases - 300K QPS for timeline - on write use fan-out process to copy data for each use-case - timeline cache in Redis - when you tweet and you have 200 followers there will be 200 inserts to each follower timeline
  27. Timelines at scalecore architecture - Hadoop for batch compute and recommendation - code heavily instrumented (load times, latencies, etc.) - uses Cassandra, but moving off from it due to read times
  28. More infoSlides - http://qconsf.com/sf2012Videos - http://www.infoq.com/

×