Allan Beaufour: Growing Up

GROWING UP TIMESOPEN - 2012/10/17

AGENDAWHO’S TALKINGWHAT IS CHARTBEATWHAT IS SCALEPRINCIPLES HANDLING SCALEAMAZON USAGE

WHO’S TALKING

ALLAN BEAUFOURWEARER OF MANY HATS AT CHARTBEAT, SINCE THEEARLY DAYS (2010)JOOST (ONLINE VIDEO), FROM BEGINNING TO END (RIP)FIREFOX HACKING AT NOVELL

WHAT IS CHARTBEAT

OUR 404 PAGE FOR A WHILE :)

ANALYTICS FOR THE FRONT-LINEGIVE DATA TO PEOPLE SO THAT THEY CAN TAKE(BETTER) ACTIONSDATA FOR: THE CONTENT CREATORS, NOT THE ANALYTICS TEAM THE HERE AND NOW (REAL-TIME)

CUSTOMERS

CHARTBEAT DASHBOARD FOR SOMEECARDS.COM

SMART ANALYTICSNOT JUST A PRESENTATION OF NUMBERS WE ANALYZE AS MUCH AS WE CAN FOR YOU PRESENT DATA IN ACTIONABLE FORMATWE REACH OUT TO YOU TELL YOU WHEN SOMETHING IS GOING ON

FORECASTS AND BENCHMARKS HUD/LINK POSITION SPIKE ALERTS

CHARTBEAT STATSLAUNCHED APRIL 2ND 2009> 200 AMAZON INSTANCES> 3 TB DATA STORED / MONTH> 8B AKAMAI EDGE HITS / MONTH> 100K LIVE REQS/SEC> 6M CONCURRENTS EVERY DAY

SCALE?

WHAT IS SCALE?“TO ADJUST IN AMOUNT ACCORDING TO A FIXED SCALEOR PROPORTION”... AND DOING THIS IN A WAY THAT IS (FAIL-)SAFE FAST / EFFICIENT

WEB SCALE? (SIGH)

WHY IS THE WEB DIFFERENT?TIME TO MARKET IS (POSSIBLY) SHORTPRODUCT DEVELOPMENT IS AN ITERATIVE PROCESSPOTENTIAL FOR EXPLOSIVE GROWTHPAIN POINTS ARE NOT (NECESSARILY) KNOWN

HANDLING SCALE GROWTH HANDLING SCALE IS HANDLING GROWTH - WHILE BOTH PRODUCT AND FOUNDATION EVOLVES** FOR STARTUPS

HANDLING GROWTHHANDLING GROWTH IS BEING ABLE TO REACT ANDADAPT

PRINCIPLES

NO MAGIC BULLET (SORRY)

FAILDON’T BE AFRAID TO FAIL :)JUST TRY TO MAKE SURE THAT YOU ONLY MAKE THE SAME MISTAKE ONCE FAILING DOESN’T BRING CATASTROPHIC FAILURE (YOU NOTICE THAT YOU FAILED)

CONSERVATIVE TOOL USAGEWE USE PYTHON + JAVASCRIPT (+ C) MONGODB + MYSQLRESIST TEMPTATION TO INTRODUCE NEW TOOLS ALLTHE TIME

SERVICE-BASEDEVERYTHING IS A SERVICEEVERYTHING SPEAKS JSON OVER HTTPONE SERVICE - ONE PURPOSE(AIMING AT) DEPLOYING SERVICES SEPARATELYSTRIVE FOR HORIZONTAL SCALING

MONITORING / GRAPHINGLOTSAND LOTS(AND LOTS)

BE PRAGMATICBEWARE OF NIH PLENTY OF GOOD SOFTWARE OUT THEREMOST OF THE TIME, PERFECTION ISN’T NEEDED EX: “DO WE REALLY NEED REAL-TIME?”

SUMMARYBE PREPARED TO FAILCONSERVATIVE TOOL USAGESOAMONITORING AND GRAPHINGBE PRAGMATIC

AMAZON

AMAZON SETUPWE’RE FULLY HOSTED IN AWS200+ INSTANCESLIMITED SERVICE USAGE EC2, EBS, ELB, S3, ELASTICACHEON THE RADAR (PIOPS), RDS, DYNAMODB

PERFORMANCE ISSUESPOOR NETWORK PERFORMANCE MAX AROUND 60 MBIT/S WITH OUR WORKLOAD ~40 (NETWORK-BOUND) INSTANCES FOR BEACON TRAFFICPOOR EBS PERFORMANCE HUGE IOPS SPIKES RAID 0 FOR BOTH PERFORMANCE AND SPACE

SERVICE ISSUESZONES ARE NOT REALLY ISOLATED AT LEAST ONE REGION-WIDE FAILUREELB ISSUES DO NOT SUPPORT SUDDEN BIG SPIKES ONE INCIDENT OF 50% FAIL RATE FOR DAYS NO INBOUND SECURITY GROUPS (ON NON-VPC)

INSTANCE STABILITYINSTANCES COME AND GO THIS SHOULD NOT COME AS A SURPRISE!HIGHLIGHTS THE NEED FOR FAULT TOLERANCE CONFIGURATION MANAGEMENT

AMAZON OVERALLAMAZON IS BY NO MEANS PERFECT, BUT HAS OVERALL SERVED US PRETTY WELL HAS GROWN WITH US

CONCLUSIONHANDLING SCALE (FOR A STARTUP) IS ABOUT BEINGABLE TO REACT AND ADAPTFIND AND FOLLOW (YOUR) PRINCIPLES (YOU ARE MORE THAN WELCOME TO BORROW OURS :) )CREATE AN ENVIRONMENT WHERE IT’S POSSIBLE TOFAIL GRACEFULLY

FURTHER READINGMORE CHARTBEAT TALKSJOHN ALLSPAW: RESILIENCE ENGINEERINGROBERT COOK’S TALK AT VELOCITY 2012

THANKSEMAIL: ALLAN@TWITTER: BEAUFOURSLIDES: HTTP://SLIDESHARE.NET/CHARTBEATCODE: HTTP://GITHUB.COM/CHARTBEATNEW FUN CHALLENGES: HTTP://CHARTBEAT.COM/JOBS/

http://engineering.chartbeat.com	149
https://twitter.com	6
https://si0.twimg.com	1

Allan Beaufour: Growing Up

by Chartbeat on Oct 17, 2012

Accessibility

Categories

Upload Details

Usage Rights

3 Embeds 156

Statistics