SlideShare a Scribd company logo
Sergejus Barinovas
Why                San Francisco?

Learn how others are doing at scale

Learn what problems others have

Learn does their solutions apply to us

Learn does their problems apply to us
Why              San Francisco?

Silicon Valley based companies:

  - Google               - Pinterest

  - Facebook             - Quora

  - Twitter              - tons of others...

  - Netflix
NoSQL: Past, Present, Future
Eric Brewer – author of CAP theorem

CP vs. AP but only on time-out (failure)
         ,
Real-time Web with
Real-time web
node.js – de-facto for real-time web

open connection for user and leave open for him

web sockets are great, but use fallbacks

 - mobile devices doesn't support web sockets

 - long polling, infinite frame, etc.

more companies moving to SPDY protocol
on mobile
Quora on mobile
first iPhone app

 - mobile app is like old app shipped on CD

 - hybrid application

  - native code for controls and navigation

  - HTML for viewing Q&A from the site

 - separate mobile optimized HTML layout of the web page
Quora on mobile
second Android app

 - created clone of iPhone app - failed!

 - UI natural on iPhone is alien on Android

 - bought Android devices and learned their philosophy

 - used new Google Android UI design guidelines

 - created new app with native for Android look & feel

 - users in India pay per MB, so had to optimize traffic

 - optimizations applied for iPhone app and web page
Quora on mobile
mobile first experience

 - mobile has very unique requirements

 - if you're good on mobile, you're good anywhere

 - don't use mobile app on tablets, create separate or use web
Continuous delivery
Continuous delivery
Jesse Robbins, author of Chef

infrastructure as code

 - full stack automation

 - datacenter API (for provisioning VMs, etc.)

 - infrastructure is a product and app is a customer
Continuous delivery
application as services

 - service orientation

 - software resiliency

 - deep instrumentation

dev / ops as teams

 - service owners

 - shared metrics / monitoring

 - continuous integration / deployment
Release engineering at
Release engineering at Facebook
Chuck Rossi – release engineering manager

deployment process

 - teams are not deploying to production by them selves

 - for communication during deployment IRC is used

 - if team member is not connected to IRC, release is skipped

 - BitTorrent for deployments

 - powerful app monitoring and profiling (instrumentation)
Release engineering at Facebook
deployment process

 - ability to release on subset of servers

 - very powerful feature flag mechanism by IP gender, age, …
                                             ,

 - karma points for developers with down-vote button

facebook.com

 - continuously deployed internally

 - employees always access latest facebook.com

 - easy to report bug from the internal facebook.com
Scaling
Scaling Pintereset
everything in Amazon cloud

before

 - had every possible ‘hot’ technology including MySQL,
  Cassandra, Mongo, Redis, Memcached, Membase, Elastic
  Search – FAIL

 - keep it simple, major re-architecting in late 2011
Scaling Pintereset
January 2012

 - Amazon EC2 + S3 + Akamai, ELB

 - 90 Web Engines + 50 API Engines

 - 66 sharded MySQL DBs + 66 slave replicas

 - 59 Redis

 - 51 Memcache

 - 1 Redis task queue + 25 task processors

 - sharded Solr

 - 6 engineers
Scaling Pintereset
now

- Amazon EC2 + S3 + Akamai, Level3, EdgeCast, ELB

- 180 Web Engines + 240 API Engines

- 80 sharded MySQL DBs + 80 slave replicas

- 110 Redis

- 200 Memcache

- 4 Redis task queues + 80 task processors

- sharded Solr

- 40 engineers
Scaling Pintereset
schemeless DB design

 - no foreign keys

 - no joins

 - denormalized data (id + JSON data)

 - users, user_has_boards, boards, board_has_pins, pins

 - read slaves

 - heavy use of cache for speed & better consistency

thinking of moving to their own DC
Architectural patterns
for high availability at
Architectural patterns for HA
Adrian Cockcroft – director of architecture at Netflix

architecture

 - everything in Amazon cloud in 3 availability zones

 - chaos Gorilla, latency Gorilla

 - service-based architecture, stateless micro-services

 - high attention for service resilience

 - handle dependent service unavailability or increased latency

started open-sourcing to improve quality of the code
Architectural patterns for HA
Cassandra usage

 - 2 dedicated Cassandra teams

 - over 50 Casssandra clusters, over 500 nodes, over 30 TB of
  data, biggest cluster has 72 nodes

 - most write operations, for reads Memcache layer is used

 - moved to SSD in Amazon instead of spinning disks and cache

 - for ETL: read Cassandara backup files using Hadoop

 - can scale zero-to-500 instances in 8 minutes
timelines at scale
Timelines at scale
Raffi Krikorian – director of Twiter's platform services

core architecture

 - pull (timeline & search) and push (mobile, streams) use-cases

 - 300K QPS for timeline

 - on write use fan-out process to copy data for each use-case

 - timeline cache in Redis

 - when you tweet and you have 200 followers there will be 200
   inserts to each follower timeline
Timelines at scale
core architecture

 - Hadoop for batch compute and recommendation

 - code heavily instrumented (load times, latencies, etc.)

 - uses Cassandra, but moving off from it due to read times
More info


Slides - http://qconsf.com/sf2012

Videos - http://www.infoq.com/

More Related Content

Similar to Flashback: QCon San Francisco 2012

SharePoint Framework -The future of SharePoint/ Office 365 developer ecosystem.
SharePoint Framework -The future of SharePoint/ Office 365 developer ecosystem. SharePoint Framework -The future of SharePoint/ Office 365 developer ecosystem.
SharePoint Framework -The future of SharePoint/ Office 365 developer ecosystem.
Kushan Lahiru Perera
 
Symphony Driver Essay
Symphony Driver EssaySymphony Driver Essay
Symphony Driver Essay
Angie Jorgensen
 
Famo.us - build native quality apps using html5 within a day
Famo.us - build native quality apps using html5 within a dayFamo.us - build native quality apps using html5 within a day
Famo.us - build native quality apps using html5 within a day
Debnath Sinha
 
Semantic technologies in practice - KULeuven 2016
Semantic technologies in practice - KULeuven 2016Semantic technologies in practice - KULeuven 2016
Semantic technologies in practice - KULeuven 2016
Aad Versteden
 
2014 09-12 lambda-architecture-at-indix
2014 09-12 lambda-architecture-at-indix2014 09-12 lambda-architecture-at-indix
2014 09-12 lambda-architecture-at-indix
Yu Ishikawa
 
20160201_resume_Vladimir_Chesnokov
20160201_resume_Vladimir_Chesnokov20160201_resume_Vladimir_Chesnokov
20160201_resume_Vladimir_ChesnokovVladimir Chesnokov
 
01 introduction to darwino
01   introduction to darwino01   introduction to darwino
01 introduction to darwino
darwinodb
 
The Evolution of Software for a Startup
The Evolution of Software for a Startup The Evolution of Software for a Startup
The Evolution of Software for a Startup
SARCCOM
 
AWS (Hadoop) Meetup 30.04.09
AWS (Hadoop) Meetup 30.04.09AWS (Hadoop) Meetup 30.04.09
AWS (Hadoop) Meetup 30.04.09
Chris Purrington
 
SAP TechEd 2010 highlights
SAP TechEd 2010 highlightsSAP TechEd 2010 highlights
SAP TechEd 2010 highlightsSascha Wenninger
 
Js foo - Sept 8 upload
Js foo - Sept 8 uploadJs foo - Sept 8 upload
Js foo - Sept 8 upload
Debnath Sinha
 
Opac labs overview-pr1.0
Opac labs overview-pr1.0Opac labs overview-pr1.0
Opac labs overview-pr1.0opaclabs
 
Architecting for Huper Growth and Great Engineering Culture
Architecting for Huper Growth and Great Engineering CultureArchitecting for Huper Growth and Great Engineering Culture
Architecting for Huper Growth and Great Engineering Culture
SARCCOM
 
Architecting for Hyper Growth and Great Engineering Culture
Architecting for Hyper Growth and Great Engineering CultureArchitecting for Hyper Growth and Great Engineering Culture
Architecting for Hyper Growth and Great Engineering Cultureifnu bima
 
Js foo famo.us- build native quality apps using html5 within a day
Js foo  famo.us- build native quality apps using html5 within a dayJs foo  famo.us- build native quality apps using html5 within a day
Js foo famo.us- build native quality apps using html5 within a day
Debnath Sinha
 
How Open Source Embiggens Salesforce.com
How Open Source Embiggens Salesforce.comHow Open Source Embiggens Salesforce.com
How Open Source Embiggens Salesforce.com
Salesforce Engineering
 
PHP is the King, nodejs is the Prince and Lua is the fool
PHP is the King, nodejs is the Prince and Lua is the foolPHP is the King, nodejs is the Prince and Lua is the fool
PHP is the King, nodejs is the Prince and Lua is the fool
Alessandro Cinelli (cirpo)
 

Similar to Flashback: QCon San Francisco 2012 (20)

SharePoint Framework -The future of SharePoint/ Office 365 developer ecosystem.
SharePoint Framework -The future of SharePoint/ Office 365 developer ecosystem. SharePoint Framework -The future of SharePoint/ Office 365 developer ecosystem.
SharePoint Framework -The future of SharePoint/ Office 365 developer ecosystem.
 
Symphony Driver Essay
Symphony Driver EssaySymphony Driver Essay
Symphony Driver Essay
 
Famo.us - build native quality apps using html5 within a day
Famo.us - build native quality apps using html5 within a dayFamo.us - build native quality apps using html5 within a day
Famo.us - build native quality apps using html5 within a day
 
Semantic technologies in practice - KULeuven 2016
Semantic technologies in practice - KULeuven 2016Semantic technologies in practice - KULeuven 2016
Semantic technologies in practice - KULeuven 2016
 
2014 09-12 lambda-architecture-at-indix
2014 09-12 lambda-architecture-at-indix2014 09-12 lambda-architecture-at-indix
2014 09-12 lambda-architecture-at-indix
 
AMIS OOW Review 2012 - Deel 7 - Lucas Jellema
AMIS OOW Review 2012 - Deel 7 - Lucas JellemaAMIS OOW Review 2012 - Deel 7 - Lucas Jellema
AMIS OOW Review 2012 - Deel 7 - Lucas Jellema
 
20160201_resume_Vladimir_Chesnokov
20160201_resume_Vladimir_Chesnokov20160201_resume_Vladimir_Chesnokov
20160201_resume_Vladimir_Chesnokov
 
01 introduction to darwino
01   introduction to darwino01   introduction to darwino
01 introduction to darwino
 
The Evolution of Software for a Startup
The Evolution of Software for a Startup The Evolution of Software for a Startup
The Evolution of Software for a Startup
 
AWS (Hadoop) Meetup 30.04.09
AWS (Hadoop) Meetup 30.04.09AWS (Hadoop) Meetup 30.04.09
AWS (Hadoop) Meetup 30.04.09
 
SAP TechEd 2010 highlights
SAP TechEd 2010 highlightsSAP TechEd 2010 highlights
SAP TechEd 2010 highlights
 
Js foo - Sept 8 upload
Js foo - Sept 8 uploadJs foo - Sept 8 upload
Js foo - Sept 8 upload
 
Opac labs overview-pr1.0
Opac labs overview-pr1.0Opac labs overview-pr1.0
Opac labs overview-pr1.0
 
Architecting for Huper Growth and Great Engineering Culture
Architecting for Huper Growth and Great Engineering CultureArchitecting for Huper Growth and Great Engineering Culture
Architecting for Huper Growth and Great Engineering Culture
 
Architecting for Hyper Growth and Great Engineering Culture
Architecting for Hyper Growth and Great Engineering CultureArchitecting for Hyper Growth and Great Engineering Culture
Architecting for Hyper Growth and Great Engineering Culture
 
Js foo famo.us- build native quality apps using html5 within a day
Js foo  famo.us- build native quality apps using html5 within a dayJs foo  famo.us- build native quality apps using html5 within a day
Js foo famo.us- build native quality apps using html5 within a day
 
How Open Source Embiggens Salesforce.com
How Open Source Embiggens Salesforce.comHow Open Source Embiggens Salesforce.com
How Open Source Embiggens Salesforce.com
 
Raghavendra_Guru
Raghavendra_GuruRaghavendra_Guru
Raghavendra_Guru
 
PHP is the King, nodejs is the Prince and Lua is the fool
PHP is the King, nodejs is the Prince and Lua is the foolPHP is the King, nodejs is the Prince and Lua is the fool
PHP is the King, nodejs is the Prince and Lua is the fool
 
AnilKumarT_Resume_latest
AnilKumarT_Resume_latestAnilKumarT_Resume_latest
AnilKumarT_Resume_latest
 

More from Sergejus Barinovas

Bringing Developers to the Next Level
Bringing Developers to the Next LevelBringing Developers to the Next Level
Bringing Developers to the Next Level
Sergejus Barinovas
 
True story of re architecting website for scale on windows azure
True story of re architecting website for scale on windows azureTrue story of re architecting website for scale on windows azure
True story of re architecting website for scale on windows azureSergejus Barinovas
 
Continuous Happiness by Continuous Delivery
Continuous Happiness by Continuous DeliveryContinuous Happiness by Continuous Delivery
Continuous Happiness by Continuous DeliverySergejus Barinovas
 
Intro to Big Data using Hadoop
Intro to Big Data using Hadoop Intro to Big Data using Hadoop
Intro to Big Data using Hadoop
Sergejus Barinovas
 
Optimizing ASP.NET application performance: tough but necessary
Optimizing ASP.NET application performance: tough but necessaryOptimizing ASP.NET application performance: tough but necessary
Optimizing ASP.NET application performance: tough but necessary
Sergejus Barinovas
 
Release Often Release Safely
Release Often Release SafelyRelease Often Release Safely
Release Often Release Safely
Sergejus Barinovas
 
Kaip Agile skatina gerųjų praktikų panaudojimą
Kaip Agile skatina gerųjų praktikų panaudojimąKaip Agile skatina gerųjų praktikų panaudojimą
Kaip Agile skatina gerųjų praktikų panaudojimą
Sergejus Barinovas
 
Introduction to Windows Azure Platform
Introduction to Windows Azure PlatformIntroduction to Windows Azure Platform
Introduction to Windows Azure Platform
Sergejus Barinovas
 
Web Scale with NoSQL
Web Scale with NoSQLWeb Scale with NoSQL
Web Scale with NoSQL
Sergejus Barinovas
 
Moving applications to the cloud
Moving applications to the cloudMoving applications to the cloud
Moving applications to the cloud
Sergejus Barinovas
 
NoSQL - what's that
NoSQL - what's thatNoSQL - what's that
NoSQL - what's that
Sergejus Barinovas
 
Architecting Windows Azure
Architecting Windows AzureArchitecting Windows Azure
Architecting Windows Azure
Sergejus Barinovas
 
Cloud Computing and Microsoft Azure Platform
Cloud Computing and Microsoft Azure PlatformCloud Computing and Microsoft Azure Platform
Cloud Computing and Microsoft Azure Platform
Sergejus Barinovas
 

More from Sergejus Barinovas (14)

Bringing Developers to the Next Level
Bringing Developers to the Next LevelBringing Developers to the Next Level
Bringing Developers to the Next Level
 
True story of re architecting website for scale on windows azure
True story of re architecting website for scale on windows azureTrue story of re architecting website for scale on windows azure
True story of re architecting website for scale on windows azure
 
Continuous Happiness by Continuous Delivery
Continuous Happiness by Continuous DeliveryContinuous Happiness by Continuous Delivery
Continuous Happiness by Continuous Delivery
 
Intro to Big Data using Hadoop
Intro to Big Data using Hadoop Intro to Big Data using Hadoop
Intro to Big Data using Hadoop
 
Optimizing ASP.NET application performance: tough but necessary
Optimizing ASP.NET application performance: tough but necessaryOptimizing ASP.NET application performance: tough but necessary
Optimizing ASP.NET application performance: tough but necessary
 
Release Often Release Safely
Release Often Release SafelyRelease Often Release Safely
Release Often Release Safely
 
Kaip Agile skatina gerųjų praktikų panaudojimą
Kaip Agile skatina gerųjų praktikų panaudojimąKaip Agile skatina gerųjų praktikų panaudojimą
Kaip Agile skatina gerųjų praktikų panaudojimą
 
Introduction to Windows Azure Platform
Introduction to Windows Azure PlatformIntroduction to Windows Azure Platform
Introduction to Windows Azure Platform
 
Web Scale with NoSQL
Web Scale with NoSQLWeb Scale with NoSQL
Web Scale with NoSQL
 
Moving applications to the cloud
Moving applications to the cloudMoving applications to the cloud
Moving applications to the cloud
 
NoSQL - what's that
NoSQL - what's thatNoSQL - what's that
NoSQL - what's that
 
Demystifying HTML5
Demystifying HTML5Demystifying HTML5
Demystifying HTML5
 
Architecting Windows Azure
Architecting Windows AzureArchitecting Windows Azure
Architecting Windows Azure
 
Cloud Computing and Microsoft Azure Platform
Cloud Computing and Microsoft Azure PlatformCloud Computing and Microsoft Azure Platform
Cloud Computing and Microsoft Azure Platform
 

Recently uploaded

PCI PIN Basics Webinar from the Controlcase Team
PCI PIN Basics Webinar from the Controlcase TeamPCI PIN Basics Webinar from the Controlcase Team
PCI PIN Basics Webinar from the Controlcase Team
ControlCase
 
Climate Impact of Software Testing at Nordic Testing Days
Climate Impact of Software Testing at Nordic Testing DaysClimate Impact of Software Testing at Nordic Testing Days
Climate Impact of Software Testing at Nordic Testing Days
Kari Kakkonen
 
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
SOFTTECHHUB
 
RESUME BUILDER APPLICATION Project for students
RESUME BUILDER APPLICATION Project for studentsRESUME BUILDER APPLICATION Project for students
RESUME BUILDER APPLICATION Project for students
KAMESHS29
 
How to Get CNIC Information System with Paksim Ga.pptx
How to Get CNIC Information System with Paksim Ga.pptxHow to Get CNIC Information System with Paksim Ga.pptx
How to Get CNIC Information System with Paksim Ga.pptx
danishmna97
 
A tale of scale & speed: How the US Navy is enabling software delivery from l...
A tale of scale & speed: How the US Navy is enabling software delivery from l...A tale of scale & speed: How the US Navy is enabling software delivery from l...
A tale of scale & speed: How the US Navy is enabling software delivery from l...
sonjaschweigert1
 
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...
Neo4j
 
Epistemic Interaction - tuning interfaces to provide information for AI support
Epistemic Interaction - tuning interfaces to provide information for AI supportEpistemic Interaction - tuning interfaces to provide information for AI support
Epistemic Interaction - tuning interfaces to provide information for AI support
Alan Dix
 
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Albert Hoitingh
 
みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...
みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...
みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...
名前 です男
 
GridMate - End to end testing is a critical piece to ensure quality and avoid...
GridMate - End to end testing is a critical piece to ensure quality and avoid...GridMate - End to end testing is a critical piece to ensure quality and avoid...
GridMate - End to end testing is a critical piece to ensure quality and avoid...
ThomasParaiso2
 
Large Language Model (LLM) and it’s Geospatial Applications
Large Language Model (LLM) and it’s Geospatial ApplicationsLarge Language Model (LLM) and it’s Geospatial Applications
Large Language Model (LLM) and it’s Geospatial Applications
Rohit Gautam
 
Essentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FMEEssentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FME
Safe Software
 
20 Comprehensive Checklist of Designing and Developing a Website
20 Comprehensive Checklist of Designing and Developing a Website20 Comprehensive Checklist of Designing and Developing a Website
20 Comprehensive Checklist of Designing and Developing a Website
Pixlogix Infotech
 
zkStudyClub - Reef: Fast Succinct Non-Interactive Zero-Knowledge Regex Proofs
zkStudyClub - Reef: Fast Succinct Non-Interactive Zero-Knowledge Regex ProofszkStudyClub - Reef: Fast Succinct Non-Interactive Zero-Knowledge Regex Proofs
zkStudyClub - Reef: Fast Succinct Non-Interactive Zero-Knowledge Regex Proofs
Alex Pruden
 
UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6
DianaGray10
 
20240607 QFM018 Elixir Reading List May 2024
20240607 QFM018 Elixir Reading List May 202420240607 QFM018 Elixir Reading List May 2024
20240607 QFM018 Elixir Reading List May 2024
Matthew Sinclair
 
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
Neo4j
 
20240609 QFM020 Irresponsible AI Reading List May 2024
20240609 QFM020 Irresponsible AI Reading List May 202420240609 QFM020 Irresponsible AI Reading List May 2024
20240609 QFM020 Irresponsible AI Reading List May 2024
Matthew Sinclair
 
Microsoft - Power Platform_G.Aspiotis.pdf
Microsoft - Power Platform_G.Aspiotis.pdfMicrosoft - Power Platform_G.Aspiotis.pdf
Microsoft - Power Platform_G.Aspiotis.pdf
Uni Systems S.M.S.A.
 

Recently uploaded (20)

PCI PIN Basics Webinar from the Controlcase Team
PCI PIN Basics Webinar from the Controlcase TeamPCI PIN Basics Webinar from the Controlcase Team
PCI PIN Basics Webinar from the Controlcase Team
 
Climate Impact of Software Testing at Nordic Testing Days
Climate Impact of Software Testing at Nordic Testing DaysClimate Impact of Software Testing at Nordic Testing Days
Climate Impact of Software Testing at Nordic Testing Days
 
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!
 
RESUME BUILDER APPLICATION Project for students
RESUME BUILDER APPLICATION Project for studentsRESUME BUILDER APPLICATION Project for students
RESUME BUILDER APPLICATION Project for students
 
How to Get CNIC Information System with Paksim Ga.pptx
How to Get CNIC Information System with Paksim Ga.pptxHow to Get CNIC Information System with Paksim Ga.pptx
How to Get CNIC Information System with Paksim Ga.pptx
 
A tale of scale & speed: How the US Navy is enabling software delivery from l...
A tale of scale & speed: How the US Navy is enabling software delivery from l...A tale of scale & speed: How the US Navy is enabling software delivery from l...
A tale of scale & speed: How the US Navy is enabling software delivery from l...
 
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...
 
Epistemic Interaction - tuning interfaces to provide information for AI support
Epistemic Interaction - tuning interfaces to provide information for AI supportEpistemic Interaction - tuning interfaces to provide information for AI support
Epistemic Interaction - tuning interfaces to provide information for AI support
 
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024
 
みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...
みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...
みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...
 
GridMate - End to end testing is a critical piece to ensure quality and avoid...
GridMate - End to end testing is a critical piece to ensure quality and avoid...GridMate - End to end testing is a critical piece to ensure quality and avoid...
GridMate - End to end testing is a critical piece to ensure quality and avoid...
 
Large Language Model (LLM) and it’s Geospatial Applications
Large Language Model (LLM) and it’s Geospatial ApplicationsLarge Language Model (LLM) and it’s Geospatial Applications
Large Language Model (LLM) and it’s Geospatial Applications
 
Essentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FMEEssentials of Automations: The Art of Triggers and Actions in FME
Essentials of Automations: The Art of Triggers and Actions in FME
 
20 Comprehensive Checklist of Designing and Developing a Website
20 Comprehensive Checklist of Designing and Developing a Website20 Comprehensive Checklist of Designing and Developing a Website
20 Comprehensive Checklist of Designing and Developing a Website
 
zkStudyClub - Reef: Fast Succinct Non-Interactive Zero-Knowledge Regex Proofs
zkStudyClub - Reef: Fast Succinct Non-Interactive Zero-Knowledge Regex ProofszkStudyClub - Reef: Fast Succinct Non-Interactive Zero-Knowledge Regex Proofs
zkStudyClub - Reef: Fast Succinct Non-Interactive Zero-Knowledge Regex Proofs
 
UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6UiPath Test Automation using UiPath Test Suite series, part 6
UiPath Test Automation using UiPath Test Suite series, part 6
 
20240607 QFM018 Elixir Reading List May 2024
20240607 QFM018 Elixir Reading List May 202420240607 QFM018 Elixir Reading List May 2024
20240607 QFM018 Elixir Reading List May 2024
 
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
GraphSummit Singapore | Neo4j Product Vision & Roadmap - Q2 2024
 
20240609 QFM020 Irresponsible AI Reading List May 2024
20240609 QFM020 Irresponsible AI Reading List May 202420240609 QFM020 Irresponsible AI Reading List May 2024
20240609 QFM020 Irresponsible AI Reading List May 2024
 
Microsoft - Power Platform_G.Aspiotis.pdf
Microsoft - Power Platform_G.Aspiotis.pdfMicrosoft - Power Platform_G.Aspiotis.pdf
Microsoft - Power Platform_G.Aspiotis.pdf
 

Flashback: QCon San Francisco 2012

  • 2. Why San Francisco? Learn how others are doing at scale Learn what problems others have Learn does their solutions apply to us Learn does their problems apply to us
  • 3. Why San Francisco? Silicon Valley based companies: - Google - Pinterest - Facebook - Quora - Twitter - tons of others... - Netflix
  • 4. NoSQL: Past, Present, Future Eric Brewer – author of CAP theorem CP vs. AP but only on time-out (failure) ,
  • 6. Real-time web node.js – de-facto for real-time web open connection for user and leave open for him web sockets are great, but use fallbacks - mobile devices doesn't support web sockets - long polling, infinite frame, etc. more companies moving to SPDY protocol
  • 8. Quora on mobile first iPhone app - mobile app is like old app shipped on CD - hybrid application - native code for controls and navigation - HTML for viewing Q&A from the site - separate mobile optimized HTML layout of the web page
  • 9. Quora on mobile second Android app - created clone of iPhone app - failed! - UI natural on iPhone is alien on Android - bought Android devices and learned their philosophy - used new Google Android UI design guidelines - created new app with native for Android look & feel - users in India pay per MB, so had to optimize traffic - optimizations applied for iPhone app and web page
  • 10. Quora on mobile mobile first experience - mobile has very unique requirements - if you're good on mobile, you're good anywhere - don't use mobile app on tablets, create separate or use web
  • 12. Continuous delivery Jesse Robbins, author of Chef infrastructure as code - full stack automation - datacenter API (for provisioning VMs, etc.) - infrastructure is a product and app is a customer
  • 13. Continuous delivery application as services - service orientation - software resiliency - deep instrumentation dev / ops as teams - service owners - shared metrics / monitoring - continuous integration / deployment
  • 15. Release engineering at Facebook Chuck Rossi – release engineering manager deployment process - teams are not deploying to production by them selves - for communication during deployment IRC is used - if team member is not connected to IRC, release is skipped - BitTorrent for deployments - powerful app monitoring and profiling (instrumentation)
  • 16. Release engineering at Facebook deployment process - ability to release on subset of servers - very powerful feature flag mechanism by IP gender, age, … , - karma points for developers with down-vote button facebook.com - continuously deployed internally - employees always access latest facebook.com - easy to report bug from the internal facebook.com
  • 18. Scaling Pintereset everything in Amazon cloud before - had every possible ‘hot’ technology including MySQL, Cassandra, Mongo, Redis, Memcached, Membase, Elastic Search – FAIL - keep it simple, major re-architecting in late 2011
  • 19. Scaling Pintereset January 2012 - Amazon EC2 + S3 + Akamai, ELB - 90 Web Engines + 50 API Engines - 66 sharded MySQL DBs + 66 slave replicas - 59 Redis - 51 Memcache - 1 Redis task queue + 25 task processors - sharded Solr - 6 engineers
  • 20. Scaling Pintereset now - Amazon EC2 + S3 + Akamai, Level3, EdgeCast, ELB - 180 Web Engines + 240 API Engines - 80 sharded MySQL DBs + 80 slave replicas - 110 Redis - 200 Memcache - 4 Redis task queues + 80 task processors - sharded Solr - 40 engineers
  • 21. Scaling Pintereset schemeless DB design - no foreign keys - no joins - denormalized data (id + JSON data) - users, user_has_boards, boards, board_has_pins, pins - read slaves - heavy use of cache for speed & better consistency thinking of moving to their own DC
  • 23. Architectural patterns for HA Adrian Cockcroft – director of architecture at Netflix architecture - everything in Amazon cloud in 3 availability zones - chaos Gorilla, latency Gorilla - service-based architecture, stateless micro-services - high attention for service resilience - handle dependent service unavailability or increased latency started open-sourcing to improve quality of the code
  • 24. Architectural patterns for HA Cassandra usage - 2 dedicated Cassandra teams - over 50 Casssandra clusters, over 500 nodes, over 30 TB of data, biggest cluster has 72 nodes - most write operations, for reads Memcache layer is used - moved to SSD in Amazon instead of spinning disks and cache - for ETL: read Cassandara backup files using Hadoop - can scale zero-to-500 instances in 8 minutes
  • 26. Timelines at scale Raffi Krikorian – director of Twiter's platform services core architecture - pull (timeline & search) and push (mobile, streams) use-cases - 300K QPS for timeline - on write use fan-out process to copy data for each use-case - timeline cache in Redis - when you tweet and you have 200 followers there will be 200 inserts to each follower timeline
  • 27. Timelines at scale core architecture - Hadoop for batch compute and recommendation - code heavily instrumented (load times, latencies, etc.) - uses Cassandra, but moving off from it due to read times
  • 28. More info Slides - http://qconsf.com/sf2012 Videos - http://www.infoq.com/