Your SlideShare is downloading. ×
0
Lookout Scaling Growing Pains
Lookout Scaling Growing Pains
Lookout Scaling Growing Pains
Lookout Scaling Growing Pains
Lookout Scaling Growing Pains
Lookout Scaling Growing Pains
Lookout Scaling Growing Pains
Lookout Scaling Growing Pains
Lookout Scaling Growing Pains
Lookout Scaling Growing Pains
Lookout Scaling Growing Pains
Lookout Scaling Growing Pains
Lookout Scaling Growing Pains
Lookout Scaling Growing Pains
Lookout Scaling Growing Pains
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Lookout Scaling Growing Pains

891

Published on

Lookout! Growing Pains was originally presented at Lookout's Scaling for Mobile event on July 25, 2013. Kyle Barton is a Senior Software Engineer at Lookout, Inc. Kyle's talk focused on the growing …

Lookout! Growing Pains was originally presented at Lookout's Scaling for Mobile event on July 25, 2013. Kyle Barton is a Senior Software Engineer at Lookout, Inc. Kyle's talk focused on the growing pains that Lookout has faced as we've scaled and engaged more users. Lookout has grown immensely in the last year. We've doubled the size of the company—added more than 80 engineers to the team, support 45+ million users, have over 1000 machines in production, see over 125,000 QPS and more than 2.6 billion requests/month. Our analysts use Hadoop, Hive, and MySQL to interactively manipulate multibillion row tables. With that, there are bound to be some growing pains and lessons learned.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
891
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
2
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. LOOKOUT! GROWING PAINS KYLE BARTON, LOOKOUT, INC.
  • 2. THEN AND NOW • Getting a solid product off of the ground • Dealing with a growing user base • Where we are today
  • 3. GETTING OFF THE GROUND MVP
  • 4. • Try to reproduce locally with a test • Production debugging • Eye balling change sets • Synclogs HOW WE DEALT
  • 5. WE STARTED GROWING • More Users • More Traffic • More Developers • More Problems
  • 6. • Ran out of integer space for primary keys • Load spikes • Clients checked in at the same time • No way to tell clients to back off PROBLEMS
  • 7. • More code going out in each deploy • Difficult to diagnose issues after deploy • Pretty much guaranteed rollbacks on deploy PROBLEMS CONT...
  • 8. • Implemented “load shedding” • Under high load respond to client with “come back later” message • Automate with ganglia checks • Randomized client check in • Picture Backup disabled by default LOAD SPIKES
  • 9. INTEGER OVERFLOW • Created new tables • Big int primary and foreign keys • Application code to use new and old tables • New data goes to new table • Old data moved on login • Background Process to update untouched records
  • 10. GIANT DEPLOYS • Feature rollouts • Slowly enable a new features • Configuration • Ability to turn off new code without rolling back
  • 11. WHERE WE ARE TODAY
  • 12. PROBLEMS • Syncml • One endpoint to service all types of requests • Bloated request and responses • Team Structure • Communication • Too many meetings
  • 13. • Versioned RESTful APIs on existing architecture • Eventually end of life old clients that haven’t upgraded DEPRECATE SYNCML
  • 14. • Persisted chat • Smaller teams • Better documentation TEAM STRUCTURE AND COMMUNICATION
  • 15. Keep in touch with @lookout /mylookout blog.lookout.com contact@lookout.com http://bit.ly/scaling-for-mobile

×