2. Who am I?
⢠My name is Jonathan H. Wage
⢠Director of Technology at OpenSky.com
⢠Started OpenSky Nashville office
⢠Open Source Software evangelist
⢠Long time Symfony and Doctrine core
contributor
OpenSky
Tuesday, May 8, 12
3. What is OpenSky?
⢠Social discovery shopping website
⢠Select your own team of people
⢠Experts, inï¬uencers and tastemakers from the
ï¬elds of:
âfashion
âfood
âhealthy living
âhome
âkids
⢠They'll select the best products out thereâ
just for you.
OpenSky
Tuesday, May 8, 12
4. 1 Year of Business
⢠1 year of business on April 1st 2012
â1.5 million users
â100 plus high proï¬le curators
⢠Martha Stewart
⢠Alicia Silverstone
⢠The Judds
⢠Bobby Flay
⢠Cake Boss
â10 million connections
â500k in revenue a week
OpenSky
Tuesday, May 8, 12
5. Offices
⢠Headquarters in Manhattan
⢠Satellite offices across the United States
âNashville
âNew Hampshire
âPortland
âSan Francisco
OpenSky
Tuesday, May 8, 12
8. Basic System Structure
MongoDB
Secondary
Replication MongoDB
Primary
MongoDB
Secondary
DEVO OSIS
MongoDB
Text
MySQL
Slave
Replication MySQL
Master
web1 web3 web5
MySQL
Slave
hornetq hornetq hornetq
MySQL Group 1
hornetq
cluster
web2 web4 web6
varnish/load
Request nginx
balancer hornetq hornetq hornetq
failover
VIP
Group1
Group 2
hornetq
OpenSky
Tuesday, May 8, 12
9. Databases
⢠MongoDB
⢠MySQL
OpenSky
Tuesday, May 8, 12
10. MongoDB
⢠What do we store in MongoDB?
â Non transactional CMS type data
⢠Products
⢠Catalog
⢠Categories
⢠Follows
⢠Offers
⢠CMS Site Data
⢠Other misc. non mission critical data
OpenSky
Tuesday, May 8, 12
11. MySQL
⢠What do we store in MySQL?
âImportant transactional data
⢠Orders
⢠Inventory
⢠Stock items
OpenSky
Tuesday, May 8, 12
12. HornetQ
⢠Cluster of HornetQ nodes
âHornetQ runs on each web node
âDEVO sends messages to HornetQ
âOSIS consumes messages from the HornetQ
cluster and performs actions on the messages
⢠interact with third party API
⢠chunk the work and multiple messages to other queues
⢠upload images to s3
OpenSky
Tuesday, May 8, 12
13. HornetQ Failover
⢠If the local HornetQ is not available on the
web node it fails over to a VIP
⢠Protects us from losing messages if we have
an issue with a local hornetq node.
OpenSky
Tuesday, May 8, 12
14. Example
⢠Image uploads in DEVO admin
âUser uploads an image
âImage is stored in MongoDB gridfs temporarily
âSend a message to OSIS about the image
âOSIS downloads the image and sends it to Amazon
âWhen done OSIS posts back to DEVO to update the database
with the new url
âImage is served from CloudFront
OpenSky
Tuesday, May 8, 12
15. Example
⢠At OpenSky we listen to the seller.follow event
and perform other actions
âforward the event to OSIS
⢠send e-mail for the follow
⢠notify sailthru API of the user following the seller
âlog the follow to other databases like an activity
feed for the whole site
ârules engines. When actions like âfollowâ are
performed we compare the action to a database
of rules and act based on what the rule requires
OpenSky
Tuesday, May 8, 12
16. Why sit behind HornetQ?
⢠Ability to retry things when they fail
⢠Keep heavy and long running operations out
of the scope of the web request
⢠Imagine if your mail service goes down while
users are registering, they will still get the
join e-mail it will just be delayed since we
donât send the mail directly from DEVO.
Instead we simply forward the user.create
event to OSIS
OpenSky
Tuesday, May 8, 12
17. DEVO
⢠Main components that make up DEVO
âSymfony2
âDoctrine2 ORM
âDoctrine MongoDB ODM
OpenSky
Tuesday, May 8, 12
19. Separate model and persistence
âEasier to test
âDonât need connection or mock connection in
order to test model since it is just POPO(plain old
php objects)
âMore ï¬exible and portable. Make your model a
dependency with a submodule
⢠share across applications that are split up
⢠more controlled change environment for the model and
database
OpenSky
Tuesday, May 8, 12
20. Working with model
$user = $dm->getRepository('User')
->createQueryBuilder()
->field('email')->equals('jonwage@gmail.com')
->getQuery()
->getSingleResult();
$seller = $dm->getRepository('Seller')
->createQueryBuilder()
->field('slug')->equals('marthastewart')
->getQuery()
->getSingleResult();
$sellerFollow = new SellerFollow($seller, $user);
OpenSky
Tuesday, May 8, 12
21. Thin Controllers
⢠Keep controllers thin and delegate work to
PHP libraries with clean and intuitive APIs
⢠A controller action in DEVO looks something
like this:
class FollowController
{
// ...
public function follow($sellerSlug)
{
// $seller = $this->findSellerBySlug($sellerSlug);
// $user = $this->getLoggedInUser();
$this->followManager->follow($seller, $user);
}
// ...
}
OpenSky
Tuesday, May 8, 12
22. Decoupled Code
⢠Functionality is abstracted away in libraries
that are used in controllers.
class FollowManager
{
// ...
public function follow(Seller $seller, User $user)
{
// ...
}
}
OpenSky
Tuesday, May 8, 12
23. Decoupled Code
⢠Decoupled code leads to
âbetter unit testing
âeasier to understand
âeasier maintain
âevolve and add features to
âlonger life expectancy
OpenSky
Tuesday, May 8, 12
24. DEVO Events
⢠In DEVO we use events heavily for managing
the execution of our own app code but also
for communicating between systems
<service id="app.listener.name" class="AppListenerSellerFollowListener">
<tag name="kernel.event_listener" event="seller.follow" method="onSellerFollow" />
</service>
class FollowManager
{
// ...
public function follow(Seller $seller, User $user)
{
// ...
$this->dispatcher->notify(new Event($seller, 'seller.follow', array(
'user' => $user
)));
}
}
OpenSky
Tuesday, May 8, 12
25. Listening to events
⢠Now we can listen to seller.follow and
perform other actions when it happens.
â Create SellerFollowListener::onSellerFollow()
class SellerFollowListener
{
/**
* Listens to 'seller.follow'
*/
public function onSellerFollow(EventInterface $event)
{
$seller = $event->getSubject();
$user = $event['user'];
// do something
}
}
OpenSky
Tuesday, May 8, 12
26. Forwarding events to OSIS
⢠We also have a mechanism setup in DEVO to
forward certain events to OSIS
⢠Conï¬gure EventForwarder to forward the seller.follow
and seller.unfollow events
<parameter key="memoryqueue.queue.seller">jms.queue.opensky.seller</parameter>
<service id="follow.event_forwarder" class="EventForwarder" scope="container">
<tag name="kernel.event_listener" event="seller.follow" method="forward" />
<tag name="kernel.event_listener" event="seller.unfollow" method="forward" />
<argument>%memoryqueue.queue.seller%</argument>
<argument type="service" id="serializers.sellerFollower" />
<argument type="service" id="memoryqueue.client" />
</service>
OpenSky
Tuesday, May 8, 12
27. The EventForwarder
class EventForwarder
{
protected $client;
protected $queueName;
protected $serializer;
protected $logger;
public function __construct($queueName, AbstractSerializer $serializer, ClientInterface $client, LoggerInterface $logger)
{
$this->serializer = $serializer;
$this->queueName = $queueName;
$this->client = $client;
$this->logger = $logger;
}
public function forward(Event $event)
{
$headers = array(
BasicMessage::EVENT_NAME => $event->getName(),
BasicMessage::HOSTNAME => php_uname('n'),
);
if ($event->has('delay')) {
$headers['_HQ_SCHED_DELIVERY'] = (time() + $event->get('delay')) * 1000;
}
$parameters = $this->serializer->toArray($event->getSubject());
$message = new BasicMessage();
$message->setHeaders($headers);
$message->setQueueName($this->queueName);
$message->setParameters($parameters);
if ($this->logger) {
$this->logger->info(sprintf('Forwarding "%s" event to "%s"', $event->getName(), $this->queueName));
$this->logger->debug('Message parameters: '.print_r($parameters, true));
}
$this->client->send($message);
}
}
OpenSky
Tuesday, May 8, 12
29. JIRA
⢠Manage product/development requests and
workï¬ow
âManaging releases
âWhat QA needs to test in each release
âWhat a developer should be working on
OpenSky
Tuesday, May 8, 12
30. github
⢠Pull requests
âCode review/comments
âIntegration with jenkins for continuous
integration
⢠In house github
âKeep sensitive information safe and in our control
⢠passwords mainly
âAbility to deploy when github has issues
⢠git ï¬ow and project branches
OpenSky
Tuesday, May 8, 12
31. pr-nightmare
⢠Robot written in ruby by Justin Hileman
(@bobthecow)
âMonitors pull requests on github
âRuns jenkins build for pull requests when ï¬rst
created and each time it is changed and
comments on the pull request with success or
failure
âKeeps our build always stable
âpr-nightmare runs on a beast of a build server so
tests run fast and in groups so you get feedback
fast
OpenSky
Tuesday, May 8, 12
32. fabric
⢠One click deploys
âMakes deploying trivial
âGet new functionality out in to the wild fast
âHotï¬x issues quickly
⢠Example commands
$ fab staging proxy.depp
$ fab staging cron.stop
$ fab staging ref:release/3.5.1 deploy
OpenSky
Tuesday, May 8, 12
33. No downtime deploys
⢠Web nodes split in two groups
âgroup1
⢠web1
⢠web3
⢠web5
âgroup2
⢠web2
⢠web4
⢠web6
OpenSky
Tuesday, May 8, 12
34. Deploy to one group at a time
Start the deploy, build everything and distribute it to the web nodes but donât make it live
$ fab prod ref:v3.5.0 deploy.start
Pull group2 from the load balancer so it is not receiving any traffic
$ fab prod proxy.not_group1
Finish deploy on the out nodes (group1)
$ fab prod:out ref:v3.5.0 deploy.finish
Test group1 and make sure everything is stable.
Flip the groups in the load balancer so group1 with the new version starts getting traffic and group2 stops getting traffic
$ fab prod proxy.flip
Finish deploy on the out nodes (group2)
$ fab prod:out ref:v3.5.0 deploy.finish
Make all nodes live
$ fab prod proxy.all
OpenSky
Tuesday, May 8, 12
35. Depped
⢠When a deploy requires downtime we âdeppâ
the site. Basically, we show a page with
pictures of Johnny Depp.
⢠Depped:
âTo be put under the spell of Johnny Depp's
charming and beautiful disposition.
⢠Depp the site with fabric and no nodes will
receive traffic
$ fab prod proxy.depp
OpenSky
Tuesday, May 8, 12
36. Database Migrations
⢠Deploys often require a migration to the
database
âAdd new tables
âAdd new ï¬elds
âMigrate some data
âRename ï¬elds
âRemove deprecated data
âAnything else you can imagine
⢠Try to make migrations backwards compatible
to avoid downtime
⢠Eventual migrations
⢠Migrate data on read and migrate when updated.
OpenSky
Tuesday, May 8, 12
37. Database Migrations
⢠Doctrine Migrations library allows database
changes to be managed with PHP code in
github and deployed with fabric
⢠Generate a new migration in DEVO
$ ./app/console doctrine:migrations:generate
class Version20120330114559 extends AbstractMigration
{
public function up(Schema $schema)
{
}
public function down(Schema $schema)
{
}
}
OpenSky
Tuesday, May 8, 12
38. Database Migrations
⢠Add SQL in the up() and down() methods.
class Version20120330114559 extends AbstractMigration
{
public function up(Schema $schema)
{
$this->addSql('ALTER TABLE stock_items ADD forceSoldout TINYINT(1) NOT NULL DEFAULT 0');
}
public function down(Schema $schema)
{
$this->addSql('ALTER TABLE stock_items DROP COLUMN forceSoldout');
}
}
⢠down() allows you to reverse migrations
OpenSky
Tuesday, May 8, 12
39. Database Migrations
⢠Deploy migrations from the console
$ ./app/console doctrine:migrations:migrate
⢠Deploying with fabric executes migrations if
any new ones are available
OpenSky
Tuesday, May 8, 12
40. Database Migrations
⢠Our migrations live in a standalone git
repository
⢠Linked to DEVO with a submodule
⢠Allows managed database changes to be
deployed standalone from a full fabric deploy
which requires pulling a group out of the load
balancer.
OpenSky
Tuesday, May 8, 12
41. Use third party services
⢠Donât reinvent the wheel outside of your core
competency
âSailthru - transactional and marketing emails
âBraintree - credit card processing
âVendornet - supplier drop-ship system
âFulï¬llment Works - managed warehouse
âKissmetrics & Google Analytics - analytics
âChartbeat - real time statistics
OpenSky
Tuesday, May 8, 12
42. Social Integration
⢠Facebook comments
âUtilize facebook comments system instead of
rolling our own
âIntegrate with our data model via FB.api for local
comments storage
⢠Facebook timeline
âPost OpenSky actions/activity to users facebook
timeline
⢠Facebook sharing
⢠Pinterest sharing
⢠Twitter sharing
OpenSky
Tuesday, May 8, 12
43. Reporting/Data Warehouse
MongoDB
ETL
MySQL Braintree
Replication ETL
MySQL
Data
Warehouse
ETL ETL
Fulï¬llment
Vendornet
Works
Warehouse
Databases
views/procs
ï¬ight_deck
mongo data mysql slave rollups/aggregates
jetstream_mongo opensky_devo atmosphere
OpenSky
Tuesday, May 8, 12
44. ï¬ight_deck
⢠DEVO and other applications only need access
to ï¬ight_deck
âSet of stored procedures that run on cron,
updating stats, aggregates, rollups, etc.
âMySQL views to expose the data needed for
dashboards, reports and other reporting user
interfaces.
OpenSky
Tuesday, May 8, 12
45. Internal communications
âIRC
⢠Day to day most real time communication between
teams is done on IRC
âJabber
âGithub Pull Request Comments
⢠All code review is done in pull request comments
âMumble
⢠Push to talk voice chat used for ï¬re ï¬ghting and deploys
âE-Mail lists
OpenSky
Tuesday, May 8, 12
46. Mumble
⢠http://www.mumble.com
âHosted mumble servers that are very affordable
OpenSky
Tuesday, May 8, 12
47. Questions?
Jonathan H. Wage
http://twitter.com/jwage
http://github.com/jwage
Weâre hiring! jwage@opensky.com
PHP and JAVA Engineers
System Administrators
Frontend Engineers (Javascript, jQuery, HTML, CSS)
Quality Assurance (Selenium, Maven)
Application DBA (MySQL, MongoDB)
OpenSky
Tuesday, May 8, 12