Inside Picnik: How We Built Picnik (and What We Learned Along the Way)

7,424 views
6,953 views

Published on

Published in: Technology
0 Comments
12 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
7,424
On SlideShare
0
From Embeds
0
Number of Embeds
766
Actions
Shares
0
Downloads
179
Comments
0
Likes
12
Embeds 0
No embeds

No notes for slide

Inside Picnik: How We Built Picnik (and What We Learned Along the Way)

  1. Inside Picnik: How we built Picnik and what we learned along the way Mike Harrington Justin Huff Web
2
Expo,
April
3rd
2009

  2. Thanks! We couldn’t have done this without help!
  3. Traffic, show scale of our problem Our happy problem
  4. Client Flex HTML Apache Data Center MySQL Python Render Storage Amazon
  5. Fun with Flex
  6. We Flash •  Browser and OS independence •  Secure and Trusted •  Huge installed base •  Graphics and animation in your browser Flex pros and cons
  7. We Flash •  Decent tool set •  Decent image support •  ActionScript Flex pros and cons
  8. Flash isn’t Perfect •  Not Windows, Not Visual Studio •  ActionScript is not C++ •  SWF Size/Modularity •  Hardware isn’t accessible •  Adobe is our Frienemy •  Missing support for jpg Compression Flex pros and cons
  9. Flash is a control freak Flex pros and cons
  10. Forecast: Cloudy with a chance of CPU
  11. Your users are your free cloud! •  Free CPU & Storage! •  Cuts down on server round trips •  More responsive (usually) •  3rd Party APIs Directly from your client •  Flash 10 enables direct load User Cloud
  12. You get what you pay for. •  Your user is a crappy sysadmin •  Hard to debug problems you can’t see •  Heterogeneous Hardware •  Some OS Differences User Cloud
  13. Client Side Challenges •  Hard to debug problems you can’t see •  Every 3rd party API has downtime •  You get blamed for everything •  Caching madness User Cloud
  14. Dealing with nasty problems Logging •  Ask customers directly •  Find local customers •  Test, test, test • 
  15. Would
we
use
Flash
again?
 Absolutely.
  16. LAMP

  17. Linux
 YUP.


  18. Apache
 Check.
 (staGc
files)

  19. MySQL
 Yeah,
that
too.

  20. P
 Is
for
Python

  21. Python
 90%
of
Picnik
is
API.

  22. CherryPy
 •  We
chose
CherryPy
 •  RESTish
API
 •  Cheetah
for
a
small
amount
of
templaGng

  23. Ok,
that
was
boring.

  24. Fun Stuff
  25. VirtualizaGon
 XEN
 •  Dual
Quad
core
machines
 •  32GB
RAM
 •  We
virtualize
everything
 •  –  Except
DB
servers

  26. VirtualizaGon
Pros
 •  ParGGon
funcGons
 •  Matching
CPU
bound
and
IO
bound
VMs
 •  Hardware
standardizaGon
(in
theory)

  27. VirtualizaGon
Cons
 •  More
complexity
 •  Uses
more
RAM
 •  IO/CPU
inefficiencies

  28. Storage
 •  We
started
with
one
server
 –  Files
stored
locally
 –  Not
scalable

  29. Storage
 •  Switched
to
MogileFS
 –  Working
great
 –  No
dedicated
storage
 machines!

  30. Storage
 •  Added
S3
into
the
mix

  31. Storage
–
S3
 S3
is
dead
simple
to
implement.
 •  For
our
usage
paberns,
it's
expensive
 •  Picnik
generates
lots
of
temp
files
 •  Problems
keeping
up
with
deletes
 •  Made
a
decision
to
focus
on
other
things
 •  before
fixing
deletes

  32. Load
Balancers
 •  Started
using
Perlbal
 •  Bought
two
BigIPs
 –  Expensive,
but
good.
 •  Outgrew
the
BigIPs
 •  Went
with
A10
Networks
 –  A
lible
bumpy
 –  They've
had
great
support.

  33. CDNs
Rock
 •  We
went
with
Level3
 •  Cut
outbound
traffic
in
half:

  34. Rendering
 We
render
images
when
a
user
saves
 •  Render
jobs
go
into
a
queue
 •  Workers
pull
from
the
queue
 •  Workers
in
both
the
DC
and
EC2
 • 
  35. Rendering
 •  Manager
process
maintain
enough
idle
 workers
 •  Workers
in
DC
are
at
~100%
uGlizaGon
 %$@#!!!
  36. EC2
 •  Our
processing
is
elasGc
between
EC2
and
our
 own
servers
 •  We
buy
servers
in
batches

  37. Monitoring
 If
a
rack
falls
in
the
DC,
does
it
make
any
sound?

  38. Monitoring
 •  Nagios
for
Up/Down

 –  Integrated
with
automaGc
tools
 –  ~120
hosts
 –  ~1100
service
checks
 •  CacG
for
general
graphing/trending
 –  ~3700
data
sources
 –  ~2800
Graphs
 •  Smokeping
 –  Network
latency

  39. Redundancy
 I
like
sleeping
at
night
 •  Makes
maintenance
tasks
easier
 •  It
takes
some
work
to
build
 •  We
built
early
(thanks
Flickr!)
 • 
  40. War
Stories

  41. ISP
Issues
 •  We're
happy
with
both
of
our
providers.
 •  But...
 –  Denial
of
service
abacks
 –  Router
problems
 –  Power
 •  Have
several
providers
 •  Monitor
&
trend!

  42. ISP
Issues
 •  High
latency
on
a
transit
link:
 •  Cause:
High
CPU
usage
on
their
aggregaGon
 router
 •  SoluGon:
Clear
and
reconfig
our
interface

  43. Amazon
Web
Services
 •  AWS
is
preby
solid
 •  But,
it's
not
100%
 •  When
AWS
breaks,
we're
down
 –  The
worst
type
of
outage
because
you
can't
do
 anything.
 •  Watch
out
for
issues
that
neither
party
 controls
 –  The
Internet
isn't
as
reliable
point
to
point

  44. EC2
Safety
Net
 •  Bug
caused
render
Gmes
to
increase

  45. EC2
Safety
net
 •  Thats
OK!

  46. Flickr
Launch
 •  Very
slow
Flickr
API
calls
post‐launch
 •  Spent
an
enGre
day
on
phone/IM
with
Flickr
 Ops
 •  Finally
discovered
an
issue
with
NAT
and
TCP
 Gmestamps
 •  Lessons:
 –  Have
tools
to
be
able
to
dive
deep
 –  Granular
monitoring

  47. Firewalls
 •  We
had
a
pair
of
Watchguard
x1250e
firewalls.
 –  Specs:
“1.5Gbps
of
firewall
throughput”
 –  Actual
at
100Mbps:

  48. authorize.net
 ConnecGvity
issues
 •  Support
was
useless
 •  Eventually
got
to
their
NOC
 •  We
rerouted
via
my
home
DSL.
 •  Lessons:
 •  –  Access
to
technical
people
 –  Handle
failure
of
services
gracefully

  49. The
end.

  50. Thanks!
 mike@picnik.com
 jusGn@picnik.com

  51. Credits/Resources
 Talks
that
influenced
us:
 hbp://Gnyurl.com/scalingwebsites
 Images:
 hbp://www.flickr.com/photos/kimberlyfaye/2785383504/
 hbp://www.flickr.com/photos/piez/574804017/sizes/o/
 hbp://www.flickr.com/photos/unclejojo/3224365412/sizes/o/
 hbp://www.flickr.com/photos/phoenixdailyphoto/1467681879/sizes/l/
 hbp://abducGonlamp.com/
 hbp://www.flickr.com/photos/msig/2716038191/
 hbp://www.flickr.com/photos/touringcyclist/303577344/
 hbp://www.flickr.com/photos/davidt2006/1407837008/


×