Like what you've read? We're frequently hiring for a variety of engineering roles at Etsy. If you're interested, drop me a line or send me your resume: mike@etsy.com.
http://www.etsy.com/careers
Details on how we capture application data in our access and error logs, as well as how to generate quick reports and graphs from these logs.
This talk was presented at O'Reilly's Velocity Online Conference on October 26, 2011.
Web Performance Culture and Tools at EtsyMike Brittain
This talk covers some of the tools that Etsy uses for measuring performance, how to instill a culture of performance, how Etsy tracks performance wins and regressions, and where to get started if you don't have a formalized performance team in your company.
Originally presented at the Boston Web Performance Meetup on Aug 24, 2011.
Continuous Integration Testing in DjangoKevin Harvey
Continuous Integration is like having a robot that cleans up after you: it installs your dependencies, builds your project, run your tests, and reports back to you. This presentation outlines two methods for CI: Travis and Jenkins.
Details on how we capture application data in our access and error logs, as well as how to generate quick reports and graphs from these logs.
This talk was presented at O'Reilly's Velocity Online Conference on October 26, 2011.
Web Performance Culture and Tools at EtsyMike Brittain
This talk covers some of the tools that Etsy uses for measuring performance, how to instill a culture of performance, how Etsy tracks performance wins and regressions, and where to get started if you don't have a formalized performance team in your company.
Originally presented at the Boston Web Performance Meetup on Aug 24, 2011.
Continuous Integration Testing in DjangoKevin Harvey
Continuous Integration is like having a robot that cleans up after you: it installs your dependencies, builds your project, run your tests, and reports back to you. This presentation outlines two methods for CI: Travis and Jenkins.
Presentation at the Manchester Java Community, October 2016, about how to build and manage a scalable Continuous Integration / Continuous Deployment pipeline.
Towards Continuous Deployment with DjangoRoger Barnes
It's no secret that python is fantastic when it comes to rapid prototyping and development. When it comes to deploying a web application, the road to glory isn't as well paved and navigating the array of techniques and tools can be daunting.
This talk will address the advantages of continuous deployment, the success factors involved and the tools available, mainly focusing on experiences with Django web development.
Slides from the Selenium Clinic Tutorial from Eurostar 2012 hosted by Simon Stewart and Alan Richardson. The tutorial was awarded "Best Tutorial" at the conference.
The reference slides were excerpted from Alan Richardson's online WebDriver course hosted at Udemy.
http://www.udemy.com/selenium-2-webdriver-basics-with-java/
After years of promoting PHPUnit I still hear it's hard to get started with unit testing. So instead of showing nice step-by-step examples on how to use PHPUnit, we're going to take an example straight from github. So I've taken the challenge to start writing tests for PHP projects that don't have unit tests in place and explain how I decide where to begin, how I approach my test strategy and how I ensure I’m covering each possible use-case (and covering the CRAP index). The goal of this presentation is to show everyone that even legacy code, spaghetti code and complex code bases can be tested. After this talk you can immediately apply my examples on your own codebase (even if it's a clean code base) and get started with testing. To follow along a basic knowledge unit testing with PHPUnit is required.
Unleashing Docker with Pipelines in Bitbucket CloudAtlassian
For those using Bitbucket Cloud, Bitbucket Pipelines makes Docker a first class citizen in your CI/CD cycle. Join Bitbucket Pipelines developer Philip Hodder to learn how containers are changing the way developers build software for the cloud, enabling faster development and safer deployments in a microservices world. He'll use practical, real-world examples to show best practices for testing, building, and pushing your Docker containers in the cloud when using Bitbucket Pipelines.
Philip Hodder, Developer, Atlassian
The Bash Dashboard (Or: How to Use Bash for Data Analysis)Bram Adams
Tutorial on how to use basic Bash concepts and commands to analyze CSV files. It uses a real-life data set and structures the content along concrete analysis questions. Feel free to contact me with questions or suggestions!
There are many styles of code review, and it can be tough to find a process that works for your team. Do you use Git, SVN, or something more exotic? Do you prefer pre-commit reviews or pull requests? Do you branch, fork, or do everything with patches? This talk examines the various review options adopted by professional teams. We'll investigate Bitbucket pull requests, Crucible reviews, and Gerrit, discussing the pros and cons of each workflow. Then I'll show you the battle-hardened peer review process - refined over thousands of code reviews and pull requests - that Atlassian teams use to maintain the high quality expected of our products.
Tim Pettersen, Senior Developer Advocate, Atlassian
Selenide alternative in Python - Introducing Selene [SeleniumCamp 2016]Iakiv Kramarenko
Talk given on SeleniumCamp 2016 about:
- What features should a "general web test automation tool" have
- Why Selenide is the one Java
- And Selene is the other in Python
- And how to use the latter
QA Fest 2018. Adam Stasiak. React Native is Coming – the story of hybrid mobi...QAFest
Main idea of this talk is to show what technologies can be used for cross-platform mobile app development and how to deal with UI tests automation for them. I will outline set of challenges every tester and developer needs to conquer and give some tips how to solve them. During this talk I will present how to apply UI tests in React Native project using Detox framework.
Presentation at the Manchester Java Community, October 2016, about how to build and manage a scalable Continuous Integration / Continuous Deployment pipeline.
Towards Continuous Deployment with DjangoRoger Barnes
It's no secret that python is fantastic when it comes to rapid prototyping and development. When it comes to deploying a web application, the road to glory isn't as well paved and navigating the array of techniques and tools can be daunting.
This talk will address the advantages of continuous deployment, the success factors involved and the tools available, mainly focusing on experiences with Django web development.
Slides from the Selenium Clinic Tutorial from Eurostar 2012 hosted by Simon Stewart and Alan Richardson. The tutorial was awarded "Best Tutorial" at the conference.
The reference slides were excerpted from Alan Richardson's online WebDriver course hosted at Udemy.
http://www.udemy.com/selenium-2-webdriver-basics-with-java/
After years of promoting PHPUnit I still hear it's hard to get started with unit testing. So instead of showing nice step-by-step examples on how to use PHPUnit, we're going to take an example straight from github. So I've taken the challenge to start writing tests for PHP projects that don't have unit tests in place and explain how I decide where to begin, how I approach my test strategy and how I ensure I’m covering each possible use-case (and covering the CRAP index). The goal of this presentation is to show everyone that even legacy code, spaghetti code and complex code bases can be tested. After this talk you can immediately apply my examples on your own codebase (even if it's a clean code base) and get started with testing. To follow along a basic knowledge unit testing with PHPUnit is required.
Unleashing Docker with Pipelines in Bitbucket CloudAtlassian
For those using Bitbucket Cloud, Bitbucket Pipelines makes Docker a first class citizen in your CI/CD cycle. Join Bitbucket Pipelines developer Philip Hodder to learn how containers are changing the way developers build software for the cloud, enabling faster development and safer deployments in a microservices world. He'll use practical, real-world examples to show best practices for testing, building, and pushing your Docker containers in the cloud when using Bitbucket Pipelines.
Philip Hodder, Developer, Atlassian
The Bash Dashboard (Or: How to Use Bash for Data Analysis)Bram Adams
Tutorial on how to use basic Bash concepts and commands to analyze CSV files. It uses a real-life data set and structures the content along concrete analysis questions. Feel free to contact me with questions or suggestions!
There are many styles of code review, and it can be tough to find a process that works for your team. Do you use Git, SVN, or something more exotic? Do you prefer pre-commit reviews or pull requests? Do you branch, fork, or do everything with patches? This talk examines the various review options adopted by professional teams. We'll investigate Bitbucket pull requests, Crucible reviews, and Gerrit, discussing the pros and cons of each workflow. Then I'll show you the battle-hardened peer review process - refined over thousands of code reviews and pull requests - that Atlassian teams use to maintain the high quality expected of our products.
Tim Pettersen, Senior Developer Advocate, Atlassian
Selenide alternative in Python - Introducing Selene [SeleniumCamp 2016]Iakiv Kramarenko
Talk given on SeleniumCamp 2016 about:
- What features should a "general web test automation tool" have
- Why Selenide is the one Java
- And Selene is the other in Python
- And how to use the latter
QA Fest 2018. Adam Stasiak. React Native is Coming – the story of hybrid mobi...QAFest
Main idea of this talk is to show what technologies can be used for cross-platform mobile app development and how to deal with UI tests automation for them. I will outline set of challenges every tester and developer needs to conquer and give some tips how to solve them. During this talk I will present how to apply UI tests in React Native project using Detox framework.
How to Get to Second Base with Your CDNMike Brittain
Tips on how to improve how you use your CDN. Condensed from a lot of material, this talk was crammed into 20 minutes.
More info available at http://mikebrittain.com.
Continuous Deployment: The Dirty DetailsMike Brittain
Presented at ALM Summit 3 in Redmond, WA. January 2013.
Like what you've read? We're frequently hiring for a variety of engineering roles at Etsy. If you're interested, drop me a line or send me your resume: mike@etsy.com.
http://www.etsy.com/careers
The practical implementation of Continuous Delivery at Etsy, and how it enables the engineering team to build features quickly, refactor and change architecture, and respond to problems in production.
Presented at GOTO Aarhus 2012.
Like what you've read? We're frequently hiring for a variety of engineering roles at Etsy. If you're interested, drop me a line or send me your resume: mike@etsy.com.
http://www.etsy.com/careers
Principles and Practices in Continuous Deployment at EtsyMike Brittain
Presented at ALM Forum 2014.
Like what you've read? We're frequently hiring for a variety of engineering roles at Etsy. If you're interested, drop me a line or send me your resume: mike@etsy.com.
http://www.etsy.com/careers
26 Disruptive & Technology Trends 2016 - 2018Brian Solis
Introducing the “26 Disruptive Technology Trends for 2016 – 2018.” In this report, we’ll explore some of the disruptive trends that are affecting pretty much everything over the next few years at least those that I’m following. It’s not just tech, though. The report is organized by socioeconomic and technological impact.
Obviously, this is not an exhaustive list of every technology and societal trend bringing about disruption on planet Earth. What follows thought definitely affects the evolution of digital Darwinism, the evolution of society and technology and its impact on behavior, expectations and customs.
How Google AppEngine deals with digital art? how about music? a few case studies developed by Stinkdigital with Google Creative Lab and how App Engine dealt with a considerable amount of visits
Best Practices in Plugin Development (WordCamp Seattle)andrewnacin
My talk -- officially named "Y U NO CODE WELL" -- at WordCamp Seattle 2011 on best practices during plugin development. Find the video, as it provides some good context and conversation.
Building web applications?
Thinking about auto-updater?
Need to document your releases?
Then look at this presentation.
You'll likely discover another point of view on these questions.
(BDT402) Performance Profiling in Production: Analyzing Web Requests at Scale...Amazon Web Services
Code profiling gives a rich, detailed view of runtime performance. However, it's difficult to achieve in production: for even a small fraction of web requests, huge challenges in scalability, access, and ease of use appear. Despite this, Yelp profiles a nontrivial fraction of its traffic by combining Amazon EC2, Amazon EMR, and Amazon S3. Developers can search, sort, filter, and combine interesting profiles; during a site slowdown or page failure, this allows a fast diagnosis and speedy recovery. Some of our analyses run nightly, while others run in real-time via Storm topologies. This session includes our use cases for code profiling, its benefits, and the implementation of its handlers and analysis flows. We include both performance results and implementation challenges of our MapReduce and Storm jobs, including code overviews. We also touch on issues such as concurrent logging, cross-data center replication, job scheduling, and API definitions.
jQuery Makes Writing JavaScript Fun Again (for HTML5 User Group)Doris Chen
Get frustrated by cross-browser incompatibility? Hate to develop application using JavaScript? jQuery is a powerful JavaScript library that can enhance your websites regardless of your background. jQuery is fast, lean, simple and hugely expandable, enabling you to build compelling web applications quickly and easily. In this session, we will start with a quick introduction of jQuery, illustrate what’s so good about jQuery, and demonstrate step by step how to develop jQuery Ajax application efficiently with database, web services, OData, NetFlix and ASP.NET MVC. Microsoft is now shipping, supporting, and contributing to jQuery, with ASP.NET and Visual Studio. New features which will be available in the next release of jQuery such as globalization, templating and data-linking will be introduced in the session as well.
Last year, AOL adopted a new content strategy and has positioned itself as a premier destination for original content. Core to this strategy is having reusable, highly efficient and optimized common code and experiences at scale, which is where jQuery comes in. Check in with Dave Artz to see how jQuery has helped his front-end standards team tackle unique challenges like optimizing 3rd party widget performance, overriding plugin functionality, and managing dependencies and updates across 100+ sites spanning multiple back-end platforms.
This talk presents an approach to building free network services and introduces Libravatar, a Django-based project to provide a federated and Open Source alternative to the Gravatar profile image hosting service, a centralised web service used by a large number of social sites in the cloud.
Gentle introduction to Pyramid. Where it comes from, how simple it, how fast, how flexible and why the future will be pyramid shaped.
Made for pyconau 2011
Similar to Advanced Topics in Continuous Deployment (20)
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
UiPath Test Automation using UiPath Test Suite series, part 3DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 3. In this session, we will cover desktop automation along with UI automation.
Topics covered:
UI automation Introduction,
UI automation Sample
Desktop automation flow
Pradeep Chinnala, Senior Consultant Automation Developer @WonderBotz and UiPath MVP
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
UiPath Test Automation using UiPath Test Suite series, part 4DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap.
The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies.
Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques
What will you get from this session?
1. Insights into SAP testing best practices
2. Heatmap utilization for testing
3. Optimization of testing processes
4. Demo
Topics covered:
Execution from the test manager
Orchestrator execution result
Defect reporting
SAP heatmap example with demo
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Albert Hoitingh
In this session I delve into the encryption technology used in Microsoft 365 and Microsoft Purview. Including the concepts of Customer Key and Double Key Encryption.
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
Connector Corner: Automate dynamic content and events by pushing a buttonDianaGray10
Here is something new! In our next Connector Corner webinar, we will demonstrate how you can use a single workflow to:
Create a campaign using Mailchimp with merge tags/fields
Send an interactive Slack channel message (using buttons)
Have the message received by managers and peers along with a test email for review
But there’s more:
In a second workflow supporting the same use case, you’ll see:
Your campaign sent to target colleagues for approval
If the “Approve” button is clicked, a Jira/Zendesk ticket is created for the marketing design team
But—if the “Reject” button is pushed, colleagues will be alerted via Slack message
Join us to learn more about this new, human-in-the-loop capability, brought to you by Integration Service connectors.
And...
Speakers:
Akshay Agnihotri, Product Manager
Charlie Greenberg, Host
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...Ramesh Iyer
In today's fast-changing business world, Companies that adapt and embrace new ideas often need help to keep up with the competition. However, fostering a culture of innovation takes much work. It takes vision, leadership and willingness to take risks in the right proportion. Sachin Dev Duggal, co-founder of Builder.ai, has perfected the art of this balance, creating a company culture where creativity and growth are nurtured at each stage.
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
1. Advanced Topics in
Continuous Deployment
Mike Brittain
Engineering Director, Etsy
@mikebrittain mikebrittain.com/talks
2. - Config flags - this one goes to eleven.
Today’s TOPICs
credit: photobookgirl (flickr)
3. - Config flags - this one goes to eleven.
- Automated deploys - never settle for anything less
Today’s TOPICs
credit: photobookgirl (flickr)
4. - Config flags - this one goes to eleven.
- Automated deploys - never settle for anything less
- Release management - who needs it?!?
Today’s TOPICs
credit: photobookgirl (flickr)
5. - Config flags - this one goes to eleven.
- Automated deploys - never settle for anything less
- Release management - who needs it?!?
- Deploying schema changes - ‘cause everybody asks
Today’s TOPICs
credit: photobookgirl (flickr)
27. Small incremental changes to the application
New classes, methods, controllers
Graphics, stylesheets, templates
Copy/content changes
App deploys
Turning flags on, off, or % ramp up
Config deploys
30. Latent bugs and security holes
Traffic management, load shedding
Adding and removing infrastructure
!
Tweaking config flags or releasing patches.
“Operating” the site
41. Interpreted language, text files.
Opcode cache (Opcache or APC)
~100 servers (web, gearman, api)
Rsync (push, not pull)
Avoid restarts
PHP & Apache
42. Lots of remote orchestration (ssh and dsh)
Push code from a git clone to production network.
Splay to a few boxes, each splays to more.
Stage files in a temp location on prod boxes.
Local rsync (using dsh) into live docroot.
Keeping things fast
43. 100+ files opened per request.
Flushing opcode cache (or graceful restart).
Mostly harmless.
What can go wrong with this?
45. Two document roots (“yin” and “yang”)
Symbolic link to the right one
Opcache has to use path name, not inode
Atomic deploys
http://codeascraft.com/2013/07/01/atomic-deploys-at-etsy/
46. Binaries, not text files
Requires restarts
Requires search index and cache warming
Rsync (push, not pull)
Solr and JVM
47. Take boxes out of rotation, deploy, bring back up
Beware capacity management
Multiple versions running for extended period
Rollbacks are a pain (esp. when in mixed-state)
Rolling restarts
49. One live cluster, one dark cluster
Deploy to dark cluster (indexes, pre-warm, restarts)
Define search clusters in app config
Switch cluster traffic via config deploy
“Flip” and “Flop”
50. Start with a shell script.
Yours will be a unique snowflake.
89. Entire app deploy took 15 minutes.
!
4 people running the deployment
8 committers
Config deploy and Chef change deployed in parallel.
90. Optimal queue size
Normalized communication
Improved visibility
Historical record is ideal for post-mortems
Organic evolution
91. Hold up the queue (.hold)
Work the issue with the people available in #push
Additional help always available in #sysops
Buddy-system for off-hours deploys
Ops-on-call, dev-on-call
When something goes wrong?
97. Our web application is largely monolithic.
Etsy.com, Support & Back-office tools,
Developer API, Gearman (async work)
98. Etsy.com, Support & Back-office tools,
Developer API, Gearman (async work)
PHP, Apache, Memcache
Our web application is largely monolithic.
99. External “services” are not deployed with
the main application.
e.g. Databases, Search, Photo storage, Payments
100. e.g. Databases, Search, Photo storage, Payments
MYSQL
(schema changes)
SOLR, JVM
(rolling restarts)
PROXY CACHE,
FILERS, AMAZON S3
(specialized infra.)
PCI
(controlled access)
External “services” are not deployed with
the main application.
101. For every config flag, there are two states
we can support — present and future.
102. ... or past and present.
For every config flag, there are two states
we can support — present and future.
103. $cfg[‘new_search’] = array('enabled' => 'off');
!
// Meanwhile...
!
if ($cfg[‘new_search’] == ‘on’) {
# New and fancy search
$results = do_solr();
} else {
# old and boring search
$results = do_grep();
}
107. !
1. Write to both versions
2. Backfill historical data
3. Read from new version
4. Cut-off writes to old version
108. 0. Add new version to schema
1. Write to both versions
2. Backfill historical data
3. Read from new version
4. Cut-off writes to old version
109. 0. Add new version to schema
Schema change to add prefs columns to “users” table.
!
“write_prefs_to_user_prefs_table” => “on”
“write_prefs_to_users_table” => “off”
“read_prefs_from_users_table” => “off”
110. 1. Write to both versions
Write code for writing prefs to the “users” table.
!
“write_prefs_to_user_prefs_table” => “on”
“write_prefs_to_users_table” => “on”
“read_prefs_from_users_table” => “off”
111. 2. Backfill historical data
Offline process to sync existing data from “user_prefs”
to new columns in “users”
112. 3. Read from new version
Data validation tests. Ensure consistency both internally
and in production.
!
“write_prefs_to_user_prefs_table” => “on”
“write_prefs_to_users_table” => “on”
“read_prefs_from_users_table” => “staff”
!
113. 3. Read from new version
Data validation tests. Ensure consistency both internally
and in production.
!
“write_prefs_to_user_prefs_table” => “on”
“write_prefs_to_users_table” => “on”
“read_prefs_from_users_table” => “1%”
!
114. 3. Read from new version
Data validation tests. Ensure consistency both internally
and in production.
!
“write_prefs_to_user_prefs_table” => “on”
“write_prefs_to_users_table” => “on”
“read_prefs_from_users_table” => “5%”
!
115. 3. Read from new version
Data validation tests. Ensure consistency both internally
and in production.
!
“write_prefs_to_user_prefs_table” => “on”
“write_prefs_to_users_table” => “on”
“read_prefs_from_users_table” => “11%”
!
“This one goes to eleven.”
116. 3. Read from new version
Data validation tests. Ensure consistency both internally
and in production.
!
“write_prefs_to_user_prefs_table” => “on”
“write_prefs_to_users_table” => “on”
“read_prefs_from_users_table” => “on” // same as 100%
!
!
!
!
117. 4. Cut-off writes to old version
After running on the new table for a significant amount
of time, we can cut off writes to the old table.
!
“write_prefs_to_user_prefs_table” => “off”
“write_prefs_to_users_table” => “on”
“read_prefs_from_users_table” => “on”
!
118. “Branch by Astraction”
Controller Controller
Users Model
“users” (old) “user_prefs” “users”
old schema new schema
(Abstraction)
http://paulhammant.com/blog/branch_by_abstraction.html
http://continuousdelivery.com/2011/05/make-large-scale-changes-incrementally-with-branch-by-abstraction/
119. Avoid temptation of putting logic into DB
Async worker queue (Gearman)
Get good at alerting on data inconsistencies
Easier to scale out app servers that DBs
Shards limit complexity
About our database design…
120. No longer valid for the business
No longer stable, valid, or trusted code
Impacting performance or readability
We can afford to spend time
Clean up old config flags?
122. Start small. (We did.)
Automated tests and production monitoring.
Have a story around maintaining quality.
“We can always go back to the old way.”
Demonstrate value to leadership.