LOOKOUT!
GROWING PAINS
KYLE BARTON, LOOKOUT, INC.
THEN AND NOW
• Getting a solid product off of the ground
• Dealing with a growing user base
• Where we are today
GETTING OFF THE GROUND
MVP
• Try to reproduce locally
with a test
• Production debugging
• Eye balling change sets
• Synclogs
HOW WE DEALT
WE STARTED GROWING
• More Users
• More Traffic
• More Developers
• More Problems
• Ran out of integer space for
primary keys
• Load spikes
• Clients checked in at the
same time
• No way to tell clients t...
• More code going out in
each deploy
• Difficult to diagnose
issues after deploy
• Pretty much guaranteed
rollbacks on deplo...
• Implemented “load
shedding”
• Under high load
respond to client with
“come back later”
message
• Automate with ganglia
c...
INTEGER OVERFLOW
• Created new tables
• Big int primary and foreign keys
• Application code to use new and old
tables
• Ne...
GIANT DEPLOYS
• Feature rollouts
• Slowly enable a new features
• Configuration
• Ability to turn off new code without
rolli...
WHERE WE ARE TODAY
PROBLEMS
• Syncml
• One endpoint to service all types of
requests
• Bloated request and responses
• Team Structure
• Commu...
• Versioned RESTful APIs on
existing architecture
• Eventually end of life old
clients that haven’t
upgraded
DEPRECATE SYN...
• Persisted chat
• Smaller teams
• Better documentation
TEAM STRUCTURE AND COMMUNICATION
Keep in touch with
@lookout
/mylookout
blog.lookout.com
contact@lookout.com
http://bit.ly/scaling-for-mobile
Upcoming SlideShare
Loading in …5
×

Lookout Scaling Growing Pains

1,362 views

Published on

Lookout! Growing Pains was originally presented at Lookout's Scaling for Mobile event on July 25, 2013. Kyle Barton is a Senior Software Engineer at Lookout, Inc. Kyle's talk focused on the growing pains that Lookout has faced as we've scaled and engaged more users. Lookout has grown immensely in the last year. We've doubled the size of the company—added more than 80 engineers to the team, support 45+ million users, have over 1000 machines in production, see over 125,000 QPS and more than 2.6 billion requests/month. Our analysts use Hadoop, Hive, and MySQL to interactively manipulate multibillion row tables. With that, there are bound to be some growing pains and lessons learned.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,362
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
2
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Lookout Scaling Growing Pains

  1. 1. LOOKOUT! GROWING PAINS KYLE BARTON, LOOKOUT, INC.
  2. 2. THEN AND NOW • Getting a solid product off of the ground • Dealing with a growing user base • Where we are today
  3. 3. GETTING OFF THE GROUND MVP
  4. 4. • Try to reproduce locally with a test • Production debugging • Eye balling change sets • Synclogs HOW WE DEALT
  5. 5. WE STARTED GROWING • More Users • More Traffic • More Developers • More Problems
  6. 6. • Ran out of integer space for primary keys • Load spikes • Clients checked in at the same time • No way to tell clients to back off PROBLEMS
  7. 7. • More code going out in each deploy • Difficult to diagnose issues after deploy • Pretty much guaranteed rollbacks on deploy PROBLEMS CONT...
  8. 8. • Implemented “load shedding” • Under high load respond to client with “come back later” message • Automate with ganglia checks • Randomized client check in • Picture Backup disabled by default LOAD SPIKES
  9. 9. INTEGER OVERFLOW • Created new tables • Big int primary and foreign keys • Application code to use new and old tables • New data goes to new table • Old data moved on login • Background Process to update untouched records
  10. 10. GIANT DEPLOYS • Feature rollouts • Slowly enable a new features • Configuration • Ability to turn off new code without rolling back
  11. 11. WHERE WE ARE TODAY
  12. 12. PROBLEMS • Syncml • One endpoint to service all types of requests • Bloated request and responses • Team Structure • Communication • Too many meetings
  13. 13. • Versioned RESTful APIs on existing architecture • Eventually end of life old clients that haven’t upgraded DEPRECATE SYNCML
  14. 14. • Persisted chat • Smaller teams • Better documentation TEAM STRUCTURE AND COMMUNICATION
  15. 15. Keep in touch with @lookout /mylookout blog.lookout.com contact@lookout.com http://bit.ly/scaling-for-mobile

×