SlideShare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our User Agreement and Privacy Policy.
SlideShare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our Privacy Policy and User Agreement for details.
Successfully reported this slideshow.
Activate your 14 day free trial to unlock unlimited reading.
18.
WE DON’T DO THIS
def do_everything(request):
hotel_id = request.GET.hotel_id
room_number = request.GET.room_number
with room_mutex(hotel_id, room_number):
room = (session.query(Room)
.filter(Room.hotel_id == hotel_id)
.filter(Room.room_number == room_number).one())
if not room.available:
return Response("Room not available”,
template=room_template)
reservation = Reservation(client=request.client, room=room)
session.add(reservation)
room.available = False
price = # price_calculation
payment = Payment(reservation=reservation, price=price)
session.add(payment)
session.commit()
url = payment.get_psp_url()
return Redirect(url)
19.
BUT WE DO THIS
• Frontend UI
• Locking rooms
• Calculating room availability
• Temporarily locking rooms
• Payment processing
• Mail
• PDF invoice generation
47.
TECHNOLOGY
• Monitoring: nagios, ganglia, pingdom
• Email: offloaded to StrongMail
• Load-balanced read slave pool
• Feature flags
• Automated server configuration and
release with Puppet and Jenkins
48.
TECHNOLOGY
• Feature flags
• Develop on Vagrant
• Celery + RabbitMQ
• Virtual customer queue
• Big data for reporting, fraud, spam,
event recommendations
50.
TIPS
• Instrument and monitor everything
• Lean
51.
HOW BIG?
• 2Gb/day database transactions
• 3.5Tb/day social data analyzed
• 15Gb/day logs
52.
ORDER PROCESSOR
• Pub/sub queue with Cassandra and
Zookeeper
53.
PUBLISHING
Publisher
Get queue lock+last batch id
Create new batch
“process orders 10, 11, 12”
Store batch id, release lock
54.
SUBSCRIBING
Subscriber
Get my latest processed batch id
Store result
Update my latest processed batch id
55.
SCALING STORAGE
• Move to NoSQL
• Aggressively move queries to slaves
• Different indexes per slave
• Better hardware
• Most optimal tables for large and
highly-utilized datasets
56.
EMAIL ADDRESSES
• Users have many email addresses.
• Lookup by email, join to users table
57.
FIRST ATTEMPT
CREATE TABLE `user_emails` (
`id` int NOT NULL AUTO_INCREMENT,
`email_address` varchar(255) NOT NULL,
... --other columns about the user
`user_id` int, --foreign key to users
KEY (`email_address`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
61.
INDEX VS PK
• InnoDB: B+trees, O(log n)
• Known user id: index on email not
needed.
• Small win on lookup: O(1)
• Big win on not storing the index.
64.
DISQUS
• >165K messages per second
• <10ms latency
• 1.3B unique visitors
• 10B page views
• 500M users in discussions
• 3M communitios
• 25M comments
65.
ORIGINAL REALTIME BACKEND
• Python + gevent
• NginxPushStream
• Network IO: great
• CPU: choking at peaks
• <15ms latency
66.
CURRENT REALTIME BACKEND
• Go
• Handles all users
• Normal load:
3200 connections/machine/sec
• <10ms latency
• Only 10%-20% CPU
67.
Workers
CURRENT REALTIME BACKEND
Subscribed to results
Push result to user
NginxPushStream
68.
TESTING
• Test with real traffic
• Measure everything
69.
LESSONS
• Do work once, distribute results.
• Most likely to fail: your code. Don’t
reinvent. Keep team small.
• End-to-end ACKs are expensive.
Avoid.
• Understand use cases when load
testing.
• Tune architecture to scale.
70.
LEARN MORE
• Instagram
• Braintree
• highscalability.com
• VelocityConf (youtube, nov 2014 @ bcn?)