WHATs & WHYs
What you need and why you need it!
Software Architect @ eBay
Notes on Notes
• Slides are unordered
• There are always alternative tools. Concentrate on
• Not talking about technology
Someone please make this presentation beautiful
• Release Notes
You have to know exactly what is deployed and when it is
There is no exception.
• Release Manager
You need this title to give the responsibility. So you can
Can’t hire a new guy, just hand it to smartest one.
• Project Manager
Usually they will tell you what needs to be deployed.
• How do you know you write a good quality code?
Share your code with your colleagues, they will enjoy to
• Tell others that their code smells
So you have a chance to get your revenge
• Standards and best practices
There is always a better way and probably someone did it
• Measure the network bandwidth on each machine.
You will need to know it just before pulling hairs out of
your head when your application is not responding.
o volker gropp
There are many others. Choose one and go.
MHA for MySQL
• Master High Availability Manager and tools for
Tools are faster than you! They don’t get tired and don’t
There is nothing to loose since it works
when your master DB is crashed
• Hardware Balancer
o Citrix NetScaler
More features and better scalability. Costs you more.
• Software Balancer
Software load balancers seems cheaper and they perform quite
well but you need a good hardware to run it!
Online Schema Change
Who wants to shutdown the site and alter a table for
an hour, then wait for the slaves to catch up?
No one! Not any more.
Openark and Percona has many more useful tools.
Use them! Tools are they to make your life easier.
• Move old and expired data to archive
o Small dataset and index = $$ CASH $$
o Faster response, low latency = $$ CASH $$
o Maintainable database = $$ CASH $$
o Faster dumps and restores = $$ CASH $$
Don’t forget to write scripts to get it back when
No Database Dependency
• NO TRIGGERS
• NO STORED PROCEDURES
• NO VIEWS
o Nobody will know the business logic
o Impossible to learn how a piece of code is functioning
o Not easy to maintain
Forget immediately what your old-school database
teacher told you in the college.
Don’t use them.
• Cache everything
• More importantly evict everything
if you can’t evict do not cache
• Web Developers Toolbox:
o Varnish, Squid, etc.
o Memcached, Redis, Couchbase, etc.
o Hazelcast, Terracotta, Coherence, EhCache, etc.
CPU cache, disk buffer, page cache, browser cache,
DNS cache, Proxy cache, HTTP Etag, Expire Headers, ….
• Declare once execute many times
Write a manifesto file as an configuration and define the
standards. Each servers configuration will be identical.
Easy and fast to configure a new server
Very fast to change any settings
More holidays for system admins
• Don’t generate logs if you don’t read them.
At least you can save I/O on disks.
• Centralized logs enables you to search every action of a
component (user, product, sale, etc)
Writing a proper log in each application, makes you enable
to search for actions of userId:101.
Your boss will love you as long as you can read the past!
Capped collection, Elastic Search, Gelfino, Grok, Kibana,
This is 21st Century! Manual deployment is
Don’t wait anyone to ask you to do it.
Use Jenkins, Maven, Ant, Bash Script, Fabric,
Capistrano, Python, Perl or whatever you can.
Still reading this?
Go and write a script.
• Separated Test Environments
• Create Test Data
• Test more than once:
Have more than one test environment
Testing affects to number of bugs dramatically.
• Every application has to create its own test data.
o Your application, your test data. You know better and It is your
• Easy to test it on every environment
• Reduces dependencies
• No need to database syncs to test DBs
• Some users have same passwords for Facebook, Gmail and so
• Users emails, bank accounts, what they bought, what they like
are valuable data
Can be sold to third parties or can be used to threaten
• Users data is personal and it should be protected
Some employees are very curious about others, specially about
Block DB Access, Authentication, Auditing, Data Encryption
There is no “Freedom of Information Law” on your users data
• Own Machines (Leads to “but it works on my machine”
• Virtual Machines
• Dedicated Servers
Give them options and let them choose one they feel
Should be Simple and Fast!
Developers loves it!
• Basic Idea:
o Small, well defined tasks
o Solve problems and unclear points immediately
o Be a communication bridge
o Remove disturbance
o Plan ahead
o Measure performance
o Learn the limits
Scrum, Kanban, Scrumban, XP, Kaizen, Six Sigma, etc..
• Monitor everything
load, disk, network, memory, db response, cache response, data
size, counts, 404s, 500s, every little tiny bit can be important.
• Define milestones
• Watch after deployments
• Write Health Check services, checking the port not
• Analyze Trends
Forecast the load on next Christmas, campaign, holiday
• Use GIT
If you think that you don’t need it, then most probably
you are doing something wrong.
GitHub, Gitolite, Gitosis, Gitstash, Gitlist, Gitweb, etc..
Define Branching Model
Authenticate and Authorize the Access
• You need someone who can move some excel
boxes from one to another cell
• Sometimes some cells needs to be merged
Those excel cells are project plan, execution plan,
priority of tasks, risk analysis, etc.
They typically have extensive
background in development
I wish so!
• Don’t keep users waiting
• Add some actions to a queue
• Reduce Race-Conditions
Everything will be okay!
Enable / Disable
• Partially Maintenance
• Less Downtime
• Decoupled features
Enable / Disable messages for a time
while payment is working
or any feature..
Budget For Teams
• Go for lunch, take them to cinema, do some team
They will never forget
Loyalty will increase
Know each other better
Increase communication between members
• Have a half day off once a month to:
Do something fun together
Listen their problems
They need to talk each other and
to do something together!
Legacy Codes and Systems
• Laugh at them
• Make fun of them
• Delete them
• Remove them
• Burn them
Be courage about designing and creating them from
They are not so big things as you think!
• Don’t have meetings
• If you really need it
o Stand-up meetings (Less time)
o Invite very few people
o Have someone to take notes
o Define time box (30 min. max)
• Need more? Break it into small pieces
o Share outcome and result of the meeting
• Cross Functional Teams Good!
• Outsourcing Generally Bad!
• Centralization / Decentralization
Varies case by case. Just think about it before you
• Details and early optimizations blocks you to start
• People need uninterrupted time to finish their tasks
• Context Over Consistency
What makes sense here may not makes sense there*
• Spend some time with other departments
• Have a question?
• Liked it?
o Use it, share it, show it to your boss
• Didn’t like it?
o Pretend that you liked it and help me to make it better