SlideShare a Scribd company logo
1 of 50
Download to read offline
MySQL at Wikipedia
How we do relational data at the Wikimedia Foundation
Jaime Crespo
Percona Live Europe 2015
-Amsterdam, 23 Sep 2015-
2© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0
MySQL at Wikipedia
Jaime Crespo
●
Sr. Database Administrator
at Wikimedia Foundation
●
Used to work as a trainer
for Oracle (MySQL), as a
Consultant (Percona) and as
a Freelance administrator
(DBAHire.com)
© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0
MySQL at Wikipedia
3
Agenda
1. The Wikimedia Foundation 4. Reliability
2. MySQL details 5. Challenges
3. Performance & Architecture 6. Q&A
4© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0
MySQL at Wikipedia
THE WIKIMEDIA
FOUNDATION
MySQL at Wikipedia
5© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0
MySQL at Wikipedia
Wikimedia Foundation
6© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0
MySQL at Wikipedia
Some stats...
●
530-430 Million UVPM (not
counting mobile devices)
●
17-20 Billion page views per month
●
14-18K new editors per month
●
35 Million Wikipedia Articles
●
8K new Wikipedia articles per day
●
27 Million open/free media files
More stats: reportcard.wmflabs.org
7© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0
MySQL at Wikipedia
What makes us different
●
The Wikimedia Foundation is a non profit
●
Funded exclusively by donations
●
These are our principles
– Stewardship
– Shared power
– Internationalism
– Free Speech
– Independence
– Freedom and open source
– Serving every human being
– Transparency
– Accountability
https://wikimediafoundation.org/wiki/Resolution:Wikimedia_Foundation_Guiding_Principles
8© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0
MySQL at Wikipedia
Openness
●
Most companies are based around
a proprietary technologies
●
All the source code we create and
use on our infrastructure is free
software
– http://git.wikimedia.org/
●
All the configuration and provisioning
infrastructure is also freely licensed
– http://git.wikimedia.org/tree/operations%2Fpuppet.git
9© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0
MySQL at Wikipedia
Transparency & Accountability
●
All software and infrastructure changes are publicly
posted*:
– https://gerrit.wikimedia.org/r/#/q/status:merged+project:operations/puppet,n,z
– https://wikitech.wikimedia.org/wiki/Server_Admin_Log
●
Issue tracker is publicly accessible
– https://phabricator.wikimedia.org/
●
Most monitoring is publicly
accessible
*except security issues (until corrected) and private information
10© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0
MySQL at Wikipedia
Privacy
●
Obliged to respect our users'
privacy
●
SSL is enforced throughout all
services
●
We host all our code, data and
services (up to our possibilities) and do not share
it with 3rd parties
– No usage of CDNs, public clouds
11© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0
MySQL at Wikipedia
No dependency
●
Even companies using open source try to bind you
to their service
●
We provide you not only the software, but also
the data dumps and the documentation to create
your own fork of our projects
– https://dumps.wikipedia.org/
– https://wikitech.wikimedia.org
– Except user's private data
12© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0
MySQL at Wikipedia
Community Resources
●
Many contributors that are not
employees with production
server access
●
We also provide a Virtual
machine (Labs) and a shared
hosting platform (tools) with
access to database replicas open to contributors
– https://wikitech.wikimedia.org/wiki/Help:Contents
– https://wikitech.wikimedia.org/wiki/Help:Tool_Labs
13© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0
MySQL at Wikipedia
Team
●
11 people in “Technical Operations”, including 1
DBA
– There is also Labs Ops, Datacenter Ops, Fundraising
Ops, Analytics Ops, Release Engineering, Services,
Devs, Performance & many volunteers supporting us
●
We may not be the busiest site, but “there is
literally nowhere else serving as many page views
per engineer”
14© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0
MySQL at Wikipedia
MYSQL DETAILS
MySQL at Wikipedia
15© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0
MySQL at Wikipedia
What do we use MySQL for?
●
Core relational data (users, text &
file metadata, ... )
– Regular browser requests
– Editing API
●
Reliable Key-value store:
– Content of each page (revision)
●
Disk-based caching:
– Secondary caching level for parsed wikitext, formulas, etc.
●
Analytics and events (with difficulty)
●
Most internal services with database needs
16© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0
MySQL at Wikipedia
What do we not use MySQL for?
(I)
●
Restful API
– Cassandra
●
Crunched analytics
– Hadoop
●
Memory caching
– Memcache
●
Queueing
– Redis
17© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0
MySQL at Wikipedia
What do we not use MySQL for?
(II)
●
Search and logs
– Elasticsearch and logstash
●
Compression
– Pages use application-side
compression
●
File storage
– We use Swift
http://blog.wikimedia.org/2012/02/09/scaling-media-storage-at-wikimedia-with-swift/
18© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0
MySQL at Wikipedia
MySQL versions
●
Past: Facebook 5.1 fork
●
Currently finishing upgrading MySQL 5.5 to
custom MariaDB 10 package
http://blog.wikimedia.org/2013/04/22/wikipedia-adopts-mariadb/
●
Relaying on several 3rd party utilities: Percona
Xtrabackup and Toolkit, mydumper, etc.
19© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0
MySQL at Wikipedia
Why MariaDB?
●
WMF, “corporate” contributor of the MariaDB Foundation
●
In general, avoiding “lock-in” for production, but certain
features are great:
– Multi-source replication
– TokuDB
– Index statistics as static tables/histograms
– Open source pool of connections
●
Things we patch/would require from upstream/3rd party:
– Query rewriting plugin
– Delayed slave
– Max query running time
– Extended PRIMARY KEY issues
– Replication state in transactional tables
20© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0
MySQL at Wikipedia
Some MySQL stats
●
~22 Billion queries a day
– Top recorded throughput for
enwiki is 145K QPS
●
>800 wikis in 280 languages
●
99.99% availability for enwiki in
the last 6 months
●
~20TB of non-duplicate live data
●
2.5 Billion article revisions
●
95 percentile of query execution time is 332us
– (API) queries running longer than 300s are killed
21© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0
MySQL at Wikipedia
my.cnf
●
https://git.wikimedia.org/blob/operations%2FPuppet/10169911757ada824
c11ee4e3dcd214bd229f247/templates%2Fmariadb%2Fproduction.my.cnf.erb
●
Particularities
– MariaDB Pool-of-threads
(max_connections = 5000)
– charset = BINARY
– rpl_semi_sync*
– userstat=1
– innodb_buffer_pool_dump_at_startup
22© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0
MySQL at Wikipedia
PERFORMANCE &
ARCHITECTURE
MySQL at Wikipedia
23© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0
MySQL at Wikipedia
Hardware and operating systems
●
Standard x86_64 servers (several providers)
●
64-192GB of RAM
●
Mostly on HDs
– Hardware RAID controller (RAID 10)
– Currently integrating SSDs for vertical scalability
●
GNU/Linux
– Ubuntu Trusty; some machines still on Precise
– Currently Migrating to Debian Jessie
24© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0
MySQL at Wikipedia
Servers
●
1300 hosts
– ~120 varnish caches
– ~320 main applications
servers, scalers, job runners
– 140 active MySQL servers
(including support and labs services)
– 31 Elasticsearch servers
– 20 LVS
– 48 media storage frontends and backends
http://ganglia.wikimedia.org
25© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0
MySQL at Wikipedia
Mediawiki software
●
Running on Apache with
PHP-HHVM
●
Mediawiki implements its own
ORM that allows database
independency
– MySQL and sqlite are the main maintained engines
●
Read-write is split at application side
– Writes and important reads go to the master
– Most reads go to the slaves
●
Chronology is checked at application side
https://www.mediawiki.org/wiki/MediaWiki
26© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0
MySQL at Wikipedia
Caching
●
Caching reads and queuing writes
– HTTP varnish caching eliminates
9/10th of the traffic
– Table level caching (templatelinks,
externallinks) makes special pages trivial
●
Those are calculated asynchonously by redis jobs
on slaves
– HTML and unrendered wikitext is also cached and
stored on memcached/parsercache db servers
27© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0
MySQL at Wikipedia
Datacenters
●
Servers are distributed among 4 datacenters:
– Ashburn, Virginia (eqiad)
– Austin, Texas (codfw)
– Amsterdam (esams)
– San Francisco, California (ulsfo)
●
Only active for caching (passive for application
servers, for now)
http://blog.wikimedia.org/2013/01/19/wikimedia-sites-move-to-primary-data-center-in-ashburn-virginia/
28© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0
MySQL at Wikipedia
DNS-based CDN
http://blog.wikimedia.org/2014/07/11/making-wikimedia-sites-faster/
http://blog.wikimedia.org/2014/07/09/how-ripe-atlas-helped-wikipedia-users/
29© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0
MySQL at Wikipedia
MySQL Functional groups
●
“Core” Production Servers
●
External Storage
●
External Clusters
●
Miscellaneous internal services
●
Parsercache
●
Analytics
●
Labs
30© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0
MySQL at Wikipedia
MySQL Shards: Core servers
●
Most relational data: users, metadata, etc.
– s1: English Wikipedia
– s2: Large wikis
– s3: Most small wikis (~800)
– s4: Commons
– s5: Wikidata and German Wikipedia
– s6: Large wikis
– s7: Centralauth, metawiki and some large wikipedias
More details: https://noc.wikimedia.org/db.php
31© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0
MySQL at Wikipedia
MySQL Shards: External Storage
and External cluster
●
Key-value storage where the actual revision text is
– es1: Read-only Clusters
– es2-es3: Read/write cluster
●
x1: Very dynamic data / global data (mostly
writes)
– Notifications
– Extension data with very different query patterns
32© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0
MySQL at Wikipedia
MySQL Shards: Misc
●
m1-m5: Internal services databases (puppet,
phabricator, openstack, wordpress, …)
●
Parsercache (pc): secondary cache level for
rendered content
●
Analytics and research: MySQL replicas and event
logging for data analysis and statistics
– Make heavy use of multi-source replication for cross-
shard joins
33© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0
MySQL at Wikipedia
MySQL Shards: LabsDB
●
Replicas for Virtual Machines (labs) and
community contributors (tools)
●
Shared mysqls (and postrgresql) for tool users
●
Requires sanitizing
●
Challenging to administrate due to the large
difference between number of users and
resources available
34© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0
MySQL at Wikipedia
RELIABILITY
MySQL at Wikipedia
35© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0
MySQL at Wikipedia
Shard components
●
1 Master
●
2-14 slaves with traditional
replication
– Geographically distributed
over 2 datacenters
●
Semi-sync replication to
avoid data loss
36© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0
MySQL at Wikipedia
Master Failover
●
No automatic failover on the core
servers for masters
– Wikis will go to read-only mode if the
master fails
– An operator will perform the failover
(hopefully) in less than 15 minutes
●
HAProxy
– Only used for full automatic failover for misc.
services
37© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0
MySQL at Wikipedia
Slave Automatic Failover
●
Mediawiki-controlled
●
A slave is not used if:
– it is unresponsive
– Its lag is larger than the configured limit (and there
are other available slaves)
●
Other errors (or for maintenance) require human
intervention for depooling
38© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0
MySQL at Wikipedia
Load-Balancing
●
Also mediawiki-controlled
●
Each slave as a weight (0-N)
●
It can also have a role (API, slow, dump, watchlist,
recentpages, contributions, logpager)
– It helps avoiding disrupting all nodes and with buffer
pool for certain query patterns
●
Datacenters are active-active only for caches,
applications and mysql are still active-passive
39© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0
MySQL at Wikipedia
Data Recovery
●
Weekly logical backups from a spare
slave (6 month retention)
– Mostly unused except for issue
investigation
– 30-day retention on binary logs
●
~Biweekly public XML dumps
●
On node failure, recovery is handled
by cloning from another slave (rsync or xtrabackup)
●
24-hour delayed slave with all shards (multi-source,
TokuDB)
40© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0
MySQL at Wikipedia
Maintenance
●
No maintenance windows
– code deployments 24/7
●
No integrated system-
depending on the change:
– pt-online-schema-change/
online schema change
– Always enough redundancy for switchover
– Batched update
https://wikitech.wikimedia.org/wiki/Deployments
41© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0
MySQL at Wikipedia
Lessons learned about recovery
●
Avoid flopping services: STONITH
●
Chaos/monkey testing (we call it deployment schedule)
●
Backups are useless: have a faster recovery plan
– Data recovery <> service recovery
●
Avoid active-passive setups:
– Avoid failover -you won't be ready when needed
– Have redundancy and a 30% resource utilization
●
Automatize and log everything (even if run manually)
42© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0
MySQL at Wikipedia
Monitorization
●
“Ecosystem” problem: too many of them
– Ganglia: basic parameters
– Icinga: alerts
– Graphite & Graphana: custom graphs
– Logstash: centralization of logs
●
Application db errors and slow queries
– Custom DB monitoring system: “Tendril”
●
Graphs, slow queries and reports
– pt-query-digest
●
Ishmael web interface (deprecated)
43© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0
MySQL at Wikipedia
CHALLENGES
MySQL at Wikipedia
44© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0
MySQL at Wikipedia
Infrastructure and code
●
Writes are not an issue for us -reads are
– Logged users and POST requests are not cached
●
15 year old PHP application means technical debt
– Dependency on statement-based replication
– No real utf-8 support at the time
– No sql_mode set (WIP)
45© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0
MySQL at Wikipedia
Best things about MySQL
●
InnoDB is reliable
●
Easy to use
●
Fast
●
Not trying to be smart
●
Wide 3rd party support
(utilities)
46© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0
MySQL at Wikipedia
Worst things about MySQL
●
Many manual operations (provisioning,
replication, HA, partitioning)
– They have to be automated by us
– Some of them are slowly being implemented
●
Lack of proper compression (both reliable and
performant)
47© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0
MySQL at Wikipedia
Future (I)
●
SSDs and vertical scaling
●
Compression (InnoDB,
RocksDB, TokuDB?)
●
OLAP/Column based solution
for analytics
●
Fully Active-Active
over several datacenters
– Multimaster?
●
Better maintenance and recovery automation
48© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0
MySQL at Wikipedia
Future (II)
●
Integrated query analysis and
debugging (P_S?)
●
Better monitorization
– Smoke tests for data integrity,
strange states, etc.
●
10.1? 5.7? WebscaleSQL? Galera?
●
Better sanitization process (binlog processor)
●
Rearchitecture connection handling
49© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0
MySQL at Wikipedia
You can help us!
●
Apply for the DBA full time position:
http://grnh.se/0y4pxm
●
Clone our puppet repo and start sending us
patches
– Or create your own wiki-based tool on Tool-Labs
●
Join us at #wikimedia-operations and
#wikimedia-databases at Freenode
50© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0
MySQL at Wikipedia
Q&A

More Related Content

What's hot

MariaDB Server Compatibility with MySQL
MariaDB Server Compatibility with MySQLMariaDB Server Compatibility with MySQL
MariaDB Server Compatibility with MySQLColin Charles
 
Better encryption & security with MariaDB 10.1 & MySQL 5.7
Better encryption & security with MariaDB 10.1 & MySQL 5.7Better encryption & security with MariaDB 10.1 & MySQL 5.7
Better encryption & security with MariaDB 10.1 & MySQL 5.7Colin Charles
 
Lessons from database failures
Lessons from database failuresLessons from database failures
Lessons from database failuresColin Charles
 
Meet MariaDB Server 10.1 London MySQL meetup December 2015
Meet MariaDB Server 10.1 London MySQL meetup December 2015Meet MariaDB Server 10.1 London MySQL meetup December 2015
Meet MariaDB Server 10.1 London MySQL meetup December 2015Colin Charles
 
MariaDB Server & MySQL Security Essentials 2016
MariaDB Server & MySQL Security Essentials 2016MariaDB Server & MySQL Security Essentials 2016
MariaDB Server & MySQL Security Essentials 2016Colin Charles
 
Lessons from database failures
Lessons from database failuresLessons from database failures
Lessons from database failuresColin Charles
 
Webseminar: MariaDB Enterprise und MariaDB Enterprise Cluster
Webseminar: MariaDB Enterprise und MariaDB Enterprise ClusterWebseminar: MariaDB Enterprise und MariaDB Enterprise Cluster
Webseminar: MariaDB Enterprise und MariaDB Enterprise ClusterMariaDB Corporation
 
Meet MariaDB 10.1 at the Bulgaria Web Summit
Meet MariaDB 10.1 at the Bulgaria Web SummitMeet MariaDB 10.1 at the Bulgaria Web Summit
Meet MariaDB 10.1 at the Bulgaria Web SummitColin Charles
 
MySQL features missing in MariaDB Server
MySQL features missing in MariaDB ServerMySQL features missing in MariaDB Server
MySQL features missing in MariaDB ServerColin Charles
 
Distributions from the view a package
Distributions from the view a packageDistributions from the view a package
Distributions from the view a packageColin Charles
 
MySQL High Availability Solutions
MySQL High Availability SolutionsMySQL High Availability Solutions
MySQL High Availability SolutionsMydbops
 
Mysql 8 vs Mariadb 10.4 Highload++ 2019
Mysql 8 vs Mariadb 10.4 Highload++ 2019Mysql 8 vs Mariadb 10.4 Highload++ 2019
Mysql 8 vs Mariadb 10.4 Highload++ 2019Alkin Tezuysal
 
The Complete MariaDB Server tutorial
The Complete MariaDB Server tutorialThe Complete MariaDB Server tutorial
The Complete MariaDB Server tutorialColin Charles
 
The MySQL ecosystem - understanding it, not running away from it!
The MySQL ecosystem - understanding it, not running away from it! The MySQL ecosystem - understanding it, not running away from it!
The MySQL ecosystem - understanding it, not running away from it! Colin Charles
 
The MySQL Server ecosystem in 2016
The MySQL Server ecosystem in 2016The MySQL Server ecosystem in 2016
The MySQL Server ecosystem in 2016Colin Charles
 
Tips to drive maria db cluster performance for nextcloud
Tips to drive maria db cluster performance for nextcloudTips to drive maria db cluster performance for nextcloud
Tips to drive maria db cluster performance for nextcloudSeveralnines
 
MariaDB 10.1 what's new and what's coming in 10.2 - Tokyo MariaDB Meetup
MariaDB 10.1   what's new and what's coming in 10.2 - Tokyo MariaDB MeetupMariaDB 10.1   what's new and what's coming in 10.2 - Tokyo MariaDB Meetup
MariaDB 10.1 what's new and what's coming in 10.2 - Tokyo MariaDB MeetupColin Charles
 
Tuning Linux for your database FLOSSUK 2016
Tuning Linux for your database FLOSSUK 2016Tuning Linux for your database FLOSSUK 2016
Tuning Linux for your database FLOSSUK 2016Colin Charles
 
My first moments with MongoDB
My first moments with MongoDBMy first moments with MongoDB
My first moments with MongoDBColin Charles
 
MariaDB - the "new" MySQL is 5 years old and everywhere (LinuxCon Europe 2015)
MariaDB - the "new" MySQL is 5 years old and everywhere (LinuxCon Europe 2015)MariaDB - the "new" MySQL is 5 years old and everywhere (LinuxCon Europe 2015)
MariaDB - the "new" MySQL is 5 years old and everywhere (LinuxCon Europe 2015)Colin Charles
 

What's hot (20)

MariaDB Server Compatibility with MySQL
MariaDB Server Compatibility with MySQLMariaDB Server Compatibility with MySQL
MariaDB Server Compatibility with MySQL
 
Better encryption & security with MariaDB 10.1 & MySQL 5.7
Better encryption & security with MariaDB 10.1 & MySQL 5.7Better encryption & security with MariaDB 10.1 & MySQL 5.7
Better encryption & security with MariaDB 10.1 & MySQL 5.7
 
Lessons from database failures
Lessons from database failuresLessons from database failures
Lessons from database failures
 
Meet MariaDB Server 10.1 London MySQL meetup December 2015
Meet MariaDB Server 10.1 London MySQL meetup December 2015Meet MariaDB Server 10.1 London MySQL meetup December 2015
Meet MariaDB Server 10.1 London MySQL meetup December 2015
 
MariaDB Server & MySQL Security Essentials 2016
MariaDB Server & MySQL Security Essentials 2016MariaDB Server & MySQL Security Essentials 2016
MariaDB Server & MySQL Security Essentials 2016
 
Lessons from database failures
Lessons from database failuresLessons from database failures
Lessons from database failures
 
Webseminar: MariaDB Enterprise und MariaDB Enterprise Cluster
Webseminar: MariaDB Enterprise und MariaDB Enterprise ClusterWebseminar: MariaDB Enterprise und MariaDB Enterprise Cluster
Webseminar: MariaDB Enterprise und MariaDB Enterprise Cluster
 
Meet MariaDB 10.1 at the Bulgaria Web Summit
Meet MariaDB 10.1 at the Bulgaria Web SummitMeet MariaDB 10.1 at the Bulgaria Web Summit
Meet MariaDB 10.1 at the Bulgaria Web Summit
 
MySQL features missing in MariaDB Server
MySQL features missing in MariaDB ServerMySQL features missing in MariaDB Server
MySQL features missing in MariaDB Server
 
Distributions from the view a package
Distributions from the view a packageDistributions from the view a package
Distributions from the view a package
 
MySQL High Availability Solutions
MySQL High Availability SolutionsMySQL High Availability Solutions
MySQL High Availability Solutions
 
Mysql 8 vs Mariadb 10.4 Highload++ 2019
Mysql 8 vs Mariadb 10.4 Highload++ 2019Mysql 8 vs Mariadb 10.4 Highload++ 2019
Mysql 8 vs Mariadb 10.4 Highload++ 2019
 
The Complete MariaDB Server tutorial
The Complete MariaDB Server tutorialThe Complete MariaDB Server tutorial
The Complete MariaDB Server tutorial
 
The MySQL ecosystem - understanding it, not running away from it!
The MySQL ecosystem - understanding it, not running away from it! The MySQL ecosystem - understanding it, not running away from it!
The MySQL ecosystem - understanding it, not running away from it!
 
The MySQL Server ecosystem in 2016
The MySQL Server ecosystem in 2016The MySQL Server ecosystem in 2016
The MySQL Server ecosystem in 2016
 
Tips to drive maria db cluster performance for nextcloud
Tips to drive maria db cluster performance for nextcloudTips to drive maria db cluster performance for nextcloud
Tips to drive maria db cluster performance for nextcloud
 
MariaDB 10.1 what's new and what's coming in 10.2 - Tokyo MariaDB Meetup
MariaDB 10.1   what's new and what's coming in 10.2 - Tokyo MariaDB MeetupMariaDB 10.1   what's new and what's coming in 10.2 - Tokyo MariaDB Meetup
MariaDB 10.1 what's new and what's coming in 10.2 - Tokyo MariaDB Meetup
 
Tuning Linux for your database FLOSSUK 2016
Tuning Linux for your database FLOSSUK 2016Tuning Linux for your database FLOSSUK 2016
Tuning Linux for your database FLOSSUK 2016
 
My first moments with MongoDB
My first moments with MongoDBMy first moments with MongoDB
My first moments with MongoDB
 
MariaDB - the "new" MySQL is 5 years old and everywhere (LinuxCon Europe 2015)
MariaDB - the "new" MySQL is 5 years old and everywhere (LinuxCon Europe 2015)MariaDB - the "new" MySQL is 5 years old and everywhere (LinuxCon Europe 2015)
MariaDB - the "new" MySQL is 5 years old and everywhere (LinuxCon Europe 2015)
 

Similar to MySQL at Wikipedia: How we do relational data at the Wikimedia Foundation

JIO and WebViewers: interoperability for Javascript and Web Applications
JIO and WebViewers: interoperability  for Javascript and Web ApplicationsJIO and WebViewers: interoperability  for Javascript and Web Applications
JIO and WebViewers: interoperability for Javascript and Web ApplicationsXWiki
 
MySQL for Oracle DBAs
MySQL for Oracle DBAsMySQL for Oracle DBAs
MySQL for Oracle DBAsBen Krug
 
Drupal 8 and 9, Backwards Compatibility, and Drupal 8.5 update
Drupal 8 and 9, Backwards Compatibility, and Drupal 8.5 updateDrupal 8 and 9, Backwards Compatibility, and Drupal 8.5 update
Drupal 8 and 9, Backwards Compatibility, and Drupal 8.5 updateAngela Byron
 
EWD 3 Training Course Part 17: Introduction to Global Storage Databases
EWD 3 Training Course Part 17: Introduction to Global Storage DatabasesEWD 3 Training Course Part 17: Introduction to Global Storage Databases
EWD 3 Training Course Part 17: Introduction to Global Storage DatabasesRob Tweed
 
MySQL as a Document Store
MySQL as a Document StoreMySQL as a Document Store
MySQL as a Document StoreTed Wennmark
 
Docker - A high level introduction to dockers and containers
Docker - A high level introduction to dockers and containersDocker - A high level introduction to dockers and containers
Docker - A high level introduction to dockers and containersDr Ganesh Iyer
 
01282016 Aerospike-Docker webinar
01282016 Aerospike-Docker webinar01282016 Aerospike-Docker webinar
01282016 Aerospike-Docker webinarAerospike, Inc.
 
Web Technology – Web Server Setup : Chris Uriarte
Web Technology – Web Server Setup : Chris UriarteWeb Technology – Web Server Setup : Chris Uriarte
Web Technology – Web Server Setup : Chris Uriartewebhostingguy
 
Using Databases and Containers From Development to Deployment
Using Databases and Containers  From Development to DeploymentUsing Databases and Containers  From Development to Deployment
Using Databases and Containers From Development to DeploymentAerospike, Inc.
 
Introduction to WSO2 Storage Server
Introduction to WSO2 Storage Server Introduction to WSO2 Storage Server
Introduction to WSO2 Storage Server WSO2
 
Developing Real-Time Data Pipelines with Apache Kafka
Developing Real-Time Data Pipelines with Apache KafkaDeveloping Real-Time Data Pipelines with Apache Kafka
Developing Real-Time Data Pipelines with Apache KafkaJoe Stein
 
Docker enterprise Technologies
Docker enterprise TechnologiesDocker enterprise Technologies
Docker enterprise Technologiesstrikr .
 
Continuous delivery with jenkins pipelines (@devfest Vienna)
Continuous delivery with jenkins pipelines (@devfest Vienna)Continuous delivery with jenkins pipelines (@devfest Vienna)
Continuous delivery with jenkins pipelines (@devfest Vienna)Roman Pickl
 
MySQL 15 March Presentation (MySQL Tech Tour Istanbul)
MySQL 15 March Presentation (MySQL Tech Tour Istanbul)MySQL 15 March Presentation (MySQL Tech Tour Istanbul)
MySQL 15 March Presentation (MySQL Tech Tour Istanbul)Okcan Yasin Saygılı
 
Platform as a Service with Kubernetes and Mesos
Platform as a Service with Kubernetes and Mesos Platform as a Service with Kubernetes and Mesos
Platform as a Service with Kubernetes and Mesos Miguel Zuniga
 
Drupal in 5mins + Previewing Drupal 8.x
Drupal in 5mins + Previewing Drupal 8.xDrupal in 5mins + Previewing Drupal 8.x
Drupal in 5mins + Previewing Drupal 8.xWong Hoi Sing Edison
 
Real world hybrid cloud session - OpenStack DACH 2015
Real world hybrid cloud session - OpenStack DACH 2015Real world hybrid cloud session - OpenStack DACH 2015
Real world hybrid cloud session - OpenStack DACH 2015assafleb
 
Design Summit - Community and Ecosystem - John Mark Walker
Design Summit - Community and Ecosystem - John Mark Walker Design Summit - Community and Ecosystem - John Mark Walker
Design Summit - Community and Ecosystem - John Mark Walker ManageIQ
 

Similar to MySQL at Wikipedia: How we do relational data at the Wikimedia Foundation (20)

JIO and WebViewers: interoperability for Javascript and Web Applications
JIO and WebViewers: interoperability  for Javascript and Web ApplicationsJIO and WebViewers: interoperability  for Javascript and Web Applications
JIO and WebViewers: interoperability for Javascript and Web Applications
 
MySQL for Oracle DBAs
MySQL for Oracle DBAsMySQL for Oracle DBAs
MySQL for Oracle DBAs
 
Drupal 8 and 9, Backwards Compatibility, and Drupal 8.5 update
Drupal 8 and 9, Backwards Compatibility, and Drupal 8.5 updateDrupal 8 and 9, Backwards Compatibility, and Drupal 8.5 update
Drupal 8 and 9, Backwards Compatibility, and Drupal 8.5 update
 
EWD 3 Training Course Part 17: Introduction to Global Storage Databases
EWD 3 Training Course Part 17: Introduction to Global Storage DatabasesEWD 3 Training Course Part 17: Introduction to Global Storage Databases
EWD 3 Training Course Part 17: Introduction to Global Storage Databases
 
Wi Ki Presentation
Wi Ki PresentationWi Ki Presentation
Wi Ki Presentation
 
MySQL as a Document Store
MySQL as a Document StoreMySQL as a Document Store
MySQL as a Document Store
 
SD Times - Docker v2
SD Times - Docker v2SD Times - Docker v2
SD Times - Docker v2
 
Docker - A high level introduction to dockers and containers
Docker - A high level introduction to dockers and containersDocker - A high level introduction to dockers and containers
Docker - A high level introduction to dockers and containers
 
01282016 Aerospike-Docker webinar
01282016 Aerospike-Docker webinar01282016 Aerospike-Docker webinar
01282016 Aerospike-Docker webinar
 
Web Technology – Web Server Setup : Chris Uriarte
Web Technology – Web Server Setup : Chris UriarteWeb Technology – Web Server Setup : Chris Uriarte
Web Technology – Web Server Setup : Chris Uriarte
 
Using Databases and Containers From Development to Deployment
Using Databases and Containers  From Development to DeploymentUsing Databases and Containers  From Development to Deployment
Using Databases and Containers From Development to Deployment
 
Introduction to WSO2 Storage Server
Introduction to WSO2 Storage Server Introduction to WSO2 Storage Server
Introduction to WSO2 Storage Server
 
Developing Real-Time Data Pipelines with Apache Kafka
Developing Real-Time Data Pipelines with Apache KafkaDeveloping Real-Time Data Pipelines with Apache Kafka
Developing Real-Time Data Pipelines with Apache Kafka
 
Docker enterprise Technologies
Docker enterprise TechnologiesDocker enterprise Technologies
Docker enterprise Technologies
 
Continuous delivery with jenkins pipelines (@devfest Vienna)
Continuous delivery with jenkins pipelines (@devfest Vienna)Continuous delivery with jenkins pipelines (@devfest Vienna)
Continuous delivery with jenkins pipelines (@devfest Vienna)
 
MySQL 15 March Presentation (MySQL Tech Tour Istanbul)
MySQL 15 March Presentation (MySQL Tech Tour Istanbul)MySQL 15 March Presentation (MySQL Tech Tour Istanbul)
MySQL 15 March Presentation (MySQL Tech Tour Istanbul)
 
Platform as a Service with Kubernetes and Mesos
Platform as a Service with Kubernetes and Mesos Platform as a Service with Kubernetes and Mesos
Platform as a Service with Kubernetes and Mesos
 
Drupal in 5mins + Previewing Drupal 8.x
Drupal in 5mins + Previewing Drupal 8.xDrupal in 5mins + Previewing Drupal 8.x
Drupal in 5mins + Previewing Drupal 8.x
 
Real world hybrid cloud session - OpenStack DACH 2015
Real world hybrid cloud session - OpenStack DACH 2015Real world hybrid cloud session - OpenStack DACH 2015
Real world hybrid cloud session - OpenStack DACH 2015
 
Design Summit - Community and Ecosystem - John Mark Walker
Design Summit - Community and Ecosystem - John Mark Walker Design Summit - Community and Ecosystem - John Mark Walker
Design Summit - Community and Ecosystem - John Mark Walker
 

More from Jaime Crespo

Haciendo copias de seguridad de todo el conocimiento humano con python y soft...
Haciendo copias de seguridad de todo el conocimiento humano con python y soft...Haciendo copias de seguridad de todo el conocimiento humano con python y soft...
Haciendo copias de seguridad de todo el conocimiento humano con python y soft...Jaime Crespo
 
Backing up Wikipedia Databases
Backing up Wikipedia DatabasesBacking up Wikipedia Databases
Backing up Wikipedia DatabasesJaime Crespo
 
Query Optimization with MySQL 8.0 and MariaDB 10.3: The Basics
Query Optimization with MySQL 8.0 and MariaDB 10.3: The BasicsQuery Optimization with MySQL 8.0 and MariaDB 10.3: The Basics
Query Optimization with MySQL 8.0 and MariaDB 10.3: The BasicsJaime Crespo
 
Wikipedia Burgos devfest 2017
Wikipedia Burgos devfest 2017Wikipedia Burgos devfest 2017
Wikipedia Burgos devfest 2017Jaime Crespo
 
Query optimization: from 0 to 10 (and up to 5.7)
Query optimization: from 0 to 10 (and up to 5.7)Query optimization: from 0 to 10 (and up to 5.7)
Query optimization: from 0 to 10 (and up to 5.7)Jaime Crespo
 
Query Optimization with MySQL 5.7 and MariaDB 10: Even newer tricks
Query Optimization with MySQL 5.7 and MariaDB 10: Even newer tricksQuery Optimization with MySQL 5.7 and MariaDB 10: Even newer tricks
Query Optimization with MySQL 5.7 and MariaDB 10: Even newer tricksJaime Crespo
 
Query Optimization with MySQL 5.6: Old and New Tricks - Percona Live London 2013
Query Optimization with MySQL 5.6: Old and New Tricks - Percona Live London 2013Query Optimization with MySQL 5.6: Old and New Tricks - Percona Live London 2013
Query Optimization with MySQL 5.6: Old and New Tricks - Percona Live London 2013Jaime Crespo
 
Software libre para gestión de proyectos
Software libre para gestión de proyectosSoftware libre para gestión de proyectos
Software libre para gestión de proyectosJaime Crespo
 
warptalk: Experiencia de usuario y usabilidad
warptalk: Experiencia de usuario y usabilidadwarptalk: Experiencia de usuario y usabilidad
warptalk: Experiencia de usuario y usabilidadJaime Crespo
 

More from Jaime Crespo (9)

Haciendo copias de seguridad de todo el conocimiento humano con python y soft...
Haciendo copias de seguridad de todo el conocimiento humano con python y soft...Haciendo copias de seguridad de todo el conocimiento humano con python y soft...
Haciendo copias de seguridad de todo el conocimiento humano con python y soft...
 
Backing up Wikipedia Databases
Backing up Wikipedia DatabasesBacking up Wikipedia Databases
Backing up Wikipedia Databases
 
Query Optimization with MySQL 8.0 and MariaDB 10.3: The Basics
Query Optimization with MySQL 8.0 and MariaDB 10.3: The BasicsQuery Optimization with MySQL 8.0 and MariaDB 10.3: The Basics
Query Optimization with MySQL 8.0 and MariaDB 10.3: The Basics
 
Wikipedia Burgos devfest 2017
Wikipedia Burgos devfest 2017Wikipedia Burgos devfest 2017
Wikipedia Burgos devfest 2017
 
Query optimization: from 0 to 10 (and up to 5.7)
Query optimization: from 0 to 10 (and up to 5.7)Query optimization: from 0 to 10 (and up to 5.7)
Query optimization: from 0 to 10 (and up to 5.7)
 
Query Optimization with MySQL 5.7 and MariaDB 10: Even newer tricks
Query Optimization with MySQL 5.7 and MariaDB 10: Even newer tricksQuery Optimization with MySQL 5.7 and MariaDB 10: Even newer tricks
Query Optimization with MySQL 5.7 and MariaDB 10: Even newer tricks
 
Query Optimization with MySQL 5.6: Old and New Tricks - Percona Live London 2013
Query Optimization with MySQL 5.6: Old and New Tricks - Percona Live London 2013Query Optimization with MySQL 5.6: Old and New Tricks - Percona Live London 2013
Query Optimization with MySQL 5.6: Old and New Tricks - Percona Live London 2013
 
Software libre para gestión de proyectos
Software libre para gestión de proyectosSoftware libre para gestión de proyectos
Software libre para gestión de proyectos
 
warptalk: Experiencia de usuario y usabilidad
warptalk: Experiencia de usuario y usabilidadwarptalk: Experiencia de usuario y usabilidad
warptalk: Experiencia de usuario y usabilidad
 

Recently uploaded

Navigation in flutter – how to add stack, tab, and drawer navigators to your ...
Navigation in flutter – how to add stack, tab, and drawer navigators to your ...Navigation in flutter – how to add stack, tab, and drawer navigators to your ...
Navigation in flutter – how to add stack, tab, and drawer navigators to your ...Flutter Agency
 
Anypoint Code Builder - Munich MuleSoft Meetup - 16th May 2024
Anypoint Code Builder - Munich MuleSoft Meetup - 16th May 2024Anypoint Code Builder - Munich MuleSoft Meetup - 16th May 2024
Anypoint Code Builder - Munich MuleSoft Meetup - 16th May 2024MulesoftMunichMeetup
 
BusinessGPT - Security and Governance for Generative AI
BusinessGPT  - Security and Governance for Generative AIBusinessGPT  - Security and Governance for Generative AI
BusinessGPT - Security and Governance for Generative AIAGATSoftware
 
CERVED e Neo4j su una nuvola, migrazione ed evoluzione di un grafo mission cr...
CERVED e Neo4j su una nuvola, migrazione ed evoluzione di un grafo mission cr...CERVED e Neo4j su una nuvola, migrazione ed evoluzione di un grafo mission cr...
CERVED e Neo4j su una nuvola, migrazione ed evoluzione di un grafo mission cr...Neo4j
 
Abortion Pill Prices Jane Furse ](+27832195400*)[ 🏥 Women's Abortion Clinic i...
Abortion Pill Prices Jane Furse ](+27832195400*)[ 🏥 Women's Abortion Clinic i...Abortion Pill Prices Jane Furse ](+27832195400*)[ 🏥 Women's Abortion Clinic i...
Abortion Pill Prices Jane Furse ](+27832195400*)[ 🏥 Women's Abortion Clinic i...Abortion Clinic
 
Evolving Data Governance for the Real-time Streaming and AI Era
Evolving Data Governance for the Real-time Streaming and AI EraEvolving Data Governance for the Real-time Streaming and AI Era
Evolving Data Governance for the Real-time Streaming and AI Eraconfluent
 
Alluxio Monthly Webinar | Simplify Data Access for AI in Multi-Cloud
Alluxio Monthly Webinar | Simplify Data Access for AI in Multi-CloudAlluxio Monthly Webinar | Simplify Data Access for AI in Multi-Cloud
Alluxio Monthly Webinar | Simplify Data Access for AI in Multi-CloudAlluxio, Inc.
 
Encryption Recap: A Refresher on Key Concepts
Encryption Recap: A Refresher on Key ConceptsEncryption Recap: A Refresher on Key Concepts
Encryption Recap: A Refresher on Key Conceptsthomashtkim
 
Workshop - Architecting Innovative Graph Applications- GraphSummit Milan
Workshop -  Architecting Innovative Graph Applications- GraphSummit MilanWorkshop -  Architecting Innovative Graph Applications- GraphSummit Milan
Workshop - Architecting Innovative Graph Applications- GraphSummit MilanNeo4j
 
Weeding your micro service landscape.pdf
Weeding your micro service landscape.pdfWeeding your micro service landscape.pdf
Weeding your micro service landscape.pdftimtebeek1
 
UNI DI NAPOLI FEDERICO II - Il ruolo dei grafi nell'AI Conversazionale Ibrida
UNI DI NAPOLI FEDERICO II - Il ruolo dei grafi nell'AI Conversazionale IbridaUNI DI NAPOLI FEDERICO II - Il ruolo dei grafi nell'AI Conversazionale Ibrida
UNI DI NAPOLI FEDERICO II - Il ruolo dei grafi nell'AI Conversazionale IbridaNeo4j
 
From Theory to Practice: Utilizing SpiraPlan's REST API
From Theory to Practice: Utilizing SpiraPlan's REST APIFrom Theory to Practice: Utilizing SpiraPlan's REST API
From Theory to Practice: Utilizing SpiraPlan's REST APIInflectra
 
AzureNativeQumulo_HPC_Cloud_Native_Benchmarks.pdf
AzureNativeQumulo_HPC_Cloud_Native_Benchmarks.pdfAzureNativeQumulo_HPC_Cloud_Native_Benchmarks.pdf
AzureNativeQumulo_HPC_Cloud_Native_Benchmarks.pdfryanfarris8
 
Workshop: Enabling GenAI Breakthroughs with Knowledge Graphs - GraphSummit Milan
Workshop: Enabling GenAI Breakthroughs with Knowledge Graphs - GraphSummit MilanWorkshop: Enabling GenAI Breakthroughs with Knowledge Graphs - GraphSummit Milan
Workshop: Enabling GenAI Breakthroughs with Knowledge Graphs - GraphSummit MilanNeo4j
 
Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...
Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...
Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...Lisi Hocke
 

Recently uploaded (20)

Navigation in flutter – how to add stack, tab, and drawer navigators to your ...
Navigation in flutter – how to add stack, tab, and drawer navigators to your ...Navigation in flutter – how to add stack, tab, and drawer navigators to your ...
Navigation in flutter – how to add stack, tab, and drawer navigators to your ...
 
Anypoint Code Builder - Munich MuleSoft Meetup - 16th May 2024
Anypoint Code Builder - Munich MuleSoft Meetup - 16th May 2024Anypoint Code Builder - Munich MuleSoft Meetup - 16th May 2024
Anypoint Code Builder - Munich MuleSoft Meetup - 16th May 2024
 
Abortion Clinic in Bloemfontein [(+27832195400*)]🏥Safe Abortion Pills In Bloe...
Abortion Clinic in Bloemfontein [(+27832195400*)]🏥Safe Abortion Pills In Bloe...Abortion Clinic in Bloemfontein [(+27832195400*)]🏥Safe Abortion Pills In Bloe...
Abortion Clinic in Bloemfontein [(+27832195400*)]🏥Safe Abortion Pills In Bloe...
 
BusinessGPT - Security and Governance for Generative AI
BusinessGPT  - Security and Governance for Generative AIBusinessGPT  - Security and Governance for Generative AI
BusinessGPT - Security and Governance for Generative AI
 
CERVED e Neo4j su una nuvola, migrazione ed evoluzione di un grafo mission cr...
CERVED e Neo4j su una nuvola, migrazione ed evoluzione di un grafo mission cr...CERVED e Neo4j su una nuvola, migrazione ed evoluzione di un grafo mission cr...
CERVED e Neo4j su una nuvola, migrazione ed evoluzione di un grafo mission cr...
 
Abortion Pill Prices Jane Furse ](+27832195400*)[ 🏥 Women's Abortion Clinic i...
Abortion Pill Prices Jane Furse ](+27832195400*)[ 🏥 Women's Abortion Clinic i...Abortion Pill Prices Jane Furse ](+27832195400*)[ 🏥 Women's Abortion Clinic i...
Abortion Pill Prices Jane Furse ](+27832195400*)[ 🏥 Women's Abortion Clinic i...
 
Abortion Pill Prices Jane Furse ](+27832195400*)[🏥Women's Abortion Clinic in ...
Abortion Pill Prices Jane Furse ](+27832195400*)[🏥Women's Abortion Clinic in ...Abortion Pill Prices Jane Furse ](+27832195400*)[🏥Women's Abortion Clinic in ...
Abortion Pill Prices Jane Furse ](+27832195400*)[🏥Women's Abortion Clinic in ...
 
Evolving Data Governance for the Real-time Streaming and AI Era
Evolving Data Governance for the Real-time Streaming and AI EraEvolving Data Governance for the Real-time Streaming and AI Era
Evolving Data Governance for the Real-time Streaming and AI Era
 
Alluxio Monthly Webinar | Simplify Data Access for AI in Multi-Cloud
Alluxio Monthly Webinar | Simplify Data Access for AI in Multi-CloudAlluxio Monthly Webinar | Simplify Data Access for AI in Multi-Cloud
Alluxio Monthly Webinar | Simplify Data Access for AI in Multi-Cloud
 
Encryption Recap: A Refresher on Key Concepts
Encryption Recap: A Refresher on Key ConceptsEncryption Recap: A Refresher on Key Concepts
Encryption Recap: A Refresher on Key Concepts
 
Abortion Clinic In Pretoria ](+27832195400*)[ 🏥 Safe Abortion Pills in Pretor...
Abortion Clinic In Pretoria ](+27832195400*)[ 🏥 Safe Abortion Pills in Pretor...Abortion Clinic In Pretoria ](+27832195400*)[ 🏥 Safe Abortion Pills in Pretor...
Abortion Clinic In Pretoria ](+27832195400*)[ 🏥 Safe Abortion Pills in Pretor...
 
Workshop - Architecting Innovative Graph Applications- GraphSummit Milan
Workshop -  Architecting Innovative Graph Applications- GraphSummit MilanWorkshop -  Architecting Innovative Graph Applications- GraphSummit Milan
Workshop - Architecting Innovative Graph Applications- GraphSummit Milan
 
Weeding your micro service landscape.pdf
Weeding your micro service landscape.pdfWeeding your micro service landscape.pdf
Weeding your micro service landscape.pdf
 
UNI DI NAPOLI FEDERICO II - Il ruolo dei grafi nell'AI Conversazionale Ibrida
UNI DI NAPOLI FEDERICO II - Il ruolo dei grafi nell'AI Conversazionale IbridaUNI DI NAPOLI FEDERICO II - Il ruolo dei grafi nell'AI Conversazionale Ibrida
UNI DI NAPOLI FEDERICO II - Il ruolo dei grafi nell'AI Conversazionale Ibrida
 
From Theory to Practice: Utilizing SpiraPlan's REST API
From Theory to Practice: Utilizing SpiraPlan's REST APIFrom Theory to Practice: Utilizing SpiraPlan's REST API
From Theory to Practice: Utilizing SpiraPlan's REST API
 
AzureNativeQumulo_HPC_Cloud_Native_Benchmarks.pdf
AzureNativeQumulo_HPC_Cloud_Native_Benchmarks.pdfAzureNativeQumulo_HPC_Cloud_Native_Benchmarks.pdf
AzureNativeQumulo_HPC_Cloud_Native_Benchmarks.pdf
 
Abortion Clinic In Pongola ](+27832195400*)[ 🏥 Safe Abortion Pills In Pongola...
Abortion Clinic In Pongola ](+27832195400*)[ 🏥 Safe Abortion Pills In Pongola...Abortion Clinic In Pongola ](+27832195400*)[ 🏥 Safe Abortion Pills In Pongola...
Abortion Clinic In Pongola ](+27832195400*)[ 🏥 Safe Abortion Pills In Pongola...
 
Abortion Clinic In Johannesburg ](+27832195400*)[ 🏥 Safe Abortion Pills in Jo...
Abortion Clinic In Johannesburg ](+27832195400*)[ 🏥 Safe Abortion Pills in Jo...Abortion Clinic In Johannesburg ](+27832195400*)[ 🏥 Safe Abortion Pills in Jo...
Abortion Clinic In Johannesburg ](+27832195400*)[ 🏥 Safe Abortion Pills in Jo...
 
Workshop: Enabling GenAI Breakthroughs with Knowledge Graphs - GraphSummit Milan
Workshop: Enabling GenAI Breakthroughs with Knowledge Graphs - GraphSummit MilanWorkshop: Enabling GenAI Breakthroughs with Knowledge Graphs - GraphSummit Milan
Workshop: Enabling GenAI Breakthroughs with Knowledge Graphs - GraphSummit Milan
 
Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...
Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...
Team Transformation Tactics for Holistic Testing and Quality (NewCrafts Paris...
 

MySQL at Wikipedia: How we do relational data at the Wikimedia Foundation

  • 1. MySQL at Wikipedia How we do relational data at the Wikimedia Foundation Jaime Crespo Percona Live Europe 2015 -Amsterdam, 23 Sep 2015-
  • 2. 2© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0 MySQL at Wikipedia Jaime Crespo ● Sr. Database Administrator at Wikimedia Foundation ● Used to work as a trainer for Oracle (MySQL), as a Consultant (Percona) and as a Freelance administrator (DBAHire.com)
  • 3. © 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0 MySQL at Wikipedia 3 Agenda 1. The Wikimedia Foundation 4. Reliability 2. MySQL details 5. Challenges 3. Performance & Architecture 6. Q&A
  • 4. 4© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0 MySQL at Wikipedia THE WIKIMEDIA FOUNDATION MySQL at Wikipedia
  • 5. 5© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0 MySQL at Wikipedia Wikimedia Foundation
  • 6. 6© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0 MySQL at Wikipedia Some stats... ● 530-430 Million UVPM (not counting mobile devices) ● 17-20 Billion page views per month ● 14-18K new editors per month ● 35 Million Wikipedia Articles ● 8K new Wikipedia articles per day ● 27 Million open/free media files More stats: reportcard.wmflabs.org
  • 7. 7© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0 MySQL at Wikipedia What makes us different ● The Wikimedia Foundation is a non profit ● Funded exclusively by donations ● These are our principles – Stewardship – Shared power – Internationalism – Free Speech – Independence – Freedom and open source – Serving every human being – Transparency – Accountability https://wikimediafoundation.org/wiki/Resolution:Wikimedia_Foundation_Guiding_Principles
  • 8. 8© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0 MySQL at Wikipedia Openness ● Most companies are based around a proprietary technologies ● All the source code we create and use on our infrastructure is free software – http://git.wikimedia.org/ ● All the configuration and provisioning infrastructure is also freely licensed – http://git.wikimedia.org/tree/operations%2Fpuppet.git
  • 9. 9© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0 MySQL at Wikipedia Transparency & Accountability ● All software and infrastructure changes are publicly posted*: – https://gerrit.wikimedia.org/r/#/q/status:merged+project:operations/puppet,n,z – https://wikitech.wikimedia.org/wiki/Server_Admin_Log ● Issue tracker is publicly accessible – https://phabricator.wikimedia.org/ ● Most monitoring is publicly accessible *except security issues (until corrected) and private information
  • 10. 10© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0 MySQL at Wikipedia Privacy ● Obliged to respect our users' privacy ● SSL is enforced throughout all services ● We host all our code, data and services (up to our possibilities) and do not share it with 3rd parties – No usage of CDNs, public clouds
  • 11. 11© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0 MySQL at Wikipedia No dependency ● Even companies using open source try to bind you to their service ● We provide you not only the software, but also the data dumps and the documentation to create your own fork of our projects – https://dumps.wikipedia.org/ – https://wikitech.wikimedia.org – Except user's private data
  • 12. 12© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0 MySQL at Wikipedia Community Resources ● Many contributors that are not employees with production server access ● We also provide a Virtual machine (Labs) and a shared hosting platform (tools) with access to database replicas open to contributors – https://wikitech.wikimedia.org/wiki/Help:Contents – https://wikitech.wikimedia.org/wiki/Help:Tool_Labs
  • 13. 13© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0 MySQL at Wikipedia Team ● 11 people in “Technical Operations”, including 1 DBA – There is also Labs Ops, Datacenter Ops, Fundraising Ops, Analytics Ops, Release Engineering, Services, Devs, Performance & many volunteers supporting us ● We may not be the busiest site, but “there is literally nowhere else serving as many page views per engineer”
  • 14. 14© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0 MySQL at Wikipedia MYSQL DETAILS MySQL at Wikipedia
  • 15. 15© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0 MySQL at Wikipedia What do we use MySQL for? ● Core relational data (users, text & file metadata, ... ) – Regular browser requests – Editing API ● Reliable Key-value store: – Content of each page (revision) ● Disk-based caching: – Secondary caching level for parsed wikitext, formulas, etc. ● Analytics and events (with difficulty) ● Most internal services with database needs
  • 16. 16© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0 MySQL at Wikipedia What do we not use MySQL for? (I) ● Restful API – Cassandra ● Crunched analytics – Hadoop ● Memory caching – Memcache ● Queueing – Redis
  • 17. 17© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0 MySQL at Wikipedia What do we not use MySQL for? (II) ● Search and logs – Elasticsearch and logstash ● Compression – Pages use application-side compression ● File storage – We use Swift http://blog.wikimedia.org/2012/02/09/scaling-media-storage-at-wikimedia-with-swift/
  • 18. 18© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0 MySQL at Wikipedia MySQL versions ● Past: Facebook 5.1 fork ● Currently finishing upgrading MySQL 5.5 to custom MariaDB 10 package http://blog.wikimedia.org/2013/04/22/wikipedia-adopts-mariadb/ ● Relaying on several 3rd party utilities: Percona Xtrabackup and Toolkit, mydumper, etc.
  • 19. 19© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0 MySQL at Wikipedia Why MariaDB? ● WMF, “corporate” contributor of the MariaDB Foundation ● In general, avoiding “lock-in” for production, but certain features are great: – Multi-source replication – TokuDB – Index statistics as static tables/histograms – Open source pool of connections ● Things we patch/would require from upstream/3rd party: – Query rewriting plugin – Delayed slave – Max query running time – Extended PRIMARY KEY issues – Replication state in transactional tables
  • 20. 20© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0 MySQL at Wikipedia Some MySQL stats ● ~22 Billion queries a day – Top recorded throughput for enwiki is 145K QPS ● >800 wikis in 280 languages ● 99.99% availability for enwiki in the last 6 months ● ~20TB of non-duplicate live data ● 2.5 Billion article revisions ● 95 percentile of query execution time is 332us – (API) queries running longer than 300s are killed
  • 21. 21© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0 MySQL at Wikipedia my.cnf ● https://git.wikimedia.org/blob/operations%2FPuppet/10169911757ada824 c11ee4e3dcd214bd229f247/templates%2Fmariadb%2Fproduction.my.cnf.erb ● Particularities – MariaDB Pool-of-threads (max_connections = 5000) – charset = BINARY – rpl_semi_sync* – userstat=1 – innodb_buffer_pool_dump_at_startup
  • 22. 22© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0 MySQL at Wikipedia PERFORMANCE & ARCHITECTURE MySQL at Wikipedia
  • 23. 23© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0 MySQL at Wikipedia Hardware and operating systems ● Standard x86_64 servers (several providers) ● 64-192GB of RAM ● Mostly on HDs – Hardware RAID controller (RAID 10) – Currently integrating SSDs for vertical scalability ● GNU/Linux – Ubuntu Trusty; some machines still on Precise – Currently Migrating to Debian Jessie
  • 24. 24© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0 MySQL at Wikipedia Servers ● 1300 hosts – ~120 varnish caches – ~320 main applications servers, scalers, job runners – 140 active MySQL servers (including support and labs services) – 31 Elasticsearch servers – 20 LVS – 48 media storage frontends and backends http://ganglia.wikimedia.org
  • 25. 25© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0 MySQL at Wikipedia Mediawiki software ● Running on Apache with PHP-HHVM ● Mediawiki implements its own ORM that allows database independency – MySQL and sqlite are the main maintained engines ● Read-write is split at application side – Writes and important reads go to the master – Most reads go to the slaves ● Chronology is checked at application side https://www.mediawiki.org/wiki/MediaWiki
  • 26. 26© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0 MySQL at Wikipedia Caching ● Caching reads and queuing writes – HTTP varnish caching eliminates 9/10th of the traffic – Table level caching (templatelinks, externallinks) makes special pages trivial ● Those are calculated asynchonously by redis jobs on slaves – HTML and unrendered wikitext is also cached and stored on memcached/parsercache db servers
  • 27. 27© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0 MySQL at Wikipedia Datacenters ● Servers are distributed among 4 datacenters: – Ashburn, Virginia (eqiad) – Austin, Texas (codfw) – Amsterdam (esams) – San Francisco, California (ulsfo) ● Only active for caching (passive for application servers, for now) http://blog.wikimedia.org/2013/01/19/wikimedia-sites-move-to-primary-data-center-in-ashburn-virginia/
  • 28. 28© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0 MySQL at Wikipedia DNS-based CDN http://blog.wikimedia.org/2014/07/11/making-wikimedia-sites-faster/ http://blog.wikimedia.org/2014/07/09/how-ripe-atlas-helped-wikipedia-users/
  • 29. 29© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0 MySQL at Wikipedia MySQL Functional groups ● “Core” Production Servers ● External Storage ● External Clusters ● Miscellaneous internal services ● Parsercache ● Analytics ● Labs
  • 30. 30© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0 MySQL at Wikipedia MySQL Shards: Core servers ● Most relational data: users, metadata, etc. – s1: English Wikipedia – s2: Large wikis – s3: Most small wikis (~800) – s4: Commons – s5: Wikidata and German Wikipedia – s6: Large wikis – s7: Centralauth, metawiki and some large wikipedias More details: https://noc.wikimedia.org/db.php
  • 31. 31© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0 MySQL at Wikipedia MySQL Shards: External Storage and External cluster ● Key-value storage where the actual revision text is – es1: Read-only Clusters – es2-es3: Read/write cluster ● x1: Very dynamic data / global data (mostly writes) – Notifications – Extension data with very different query patterns
  • 32. 32© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0 MySQL at Wikipedia MySQL Shards: Misc ● m1-m5: Internal services databases (puppet, phabricator, openstack, wordpress, …) ● Parsercache (pc): secondary cache level for rendered content ● Analytics and research: MySQL replicas and event logging for data analysis and statistics – Make heavy use of multi-source replication for cross- shard joins
  • 33. 33© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0 MySQL at Wikipedia MySQL Shards: LabsDB ● Replicas for Virtual Machines (labs) and community contributors (tools) ● Shared mysqls (and postrgresql) for tool users ● Requires sanitizing ● Challenging to administrate due to the large difference between number of users and resources available
  • 34. 34© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0 MySQL at Wikipedia RELIABILITY MySQL at Wikipedia
  • 35. 35© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0 MySQL at Wikipedia Shard components ● 1 Master ● 2-14 slaves with traditional replication – Geographically distributed over 2 datacenters ● Semi-sync replication to avoid data loss
  • 36. 36© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0 MySQL at Wikipedia Master Failover ● No automatic failover on the core servers for masters – Wikis will go to read-only mode if the master fails – An operator will perform the failover (hopefully) in less than 15 minutes ● HAProxy – Only used for full automatic failover for misc. services
  • 37. 37© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0 MySQL at Wikipedia Slave Automatic Failover ● Mediawiki-controlled ● A slave is not used if: – it is unresponsive – Its lag is larger than the configured limit (and there are other available slaves) ● Other errors (or for maintenance) require human intervention for depooling
  • 38. 38© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0 MySQL at Wikipedia Load-Balancing ● Also mediawiki-controlled ● Each slave as a weight (0-N) ● It can also have a role (API, slow, dump, watchlist, recentpages, contributions, logpager) – It helps avoiding disrupting all nodes and with buffer pool for certain query patterns ● Datacenters are active-active only for caches, applications and mysql are still active-passive
  • 39. 39© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0 MySQL at Wikipedia Data Recovery ● Weekly logical backups from a spare slave (6 month retention) – Mostly unused except for issue investigation – 30-day retention on binary logs ● ~Biweekly public XML dumps ● On node failure, recovery is handled by cloning from another slave (rsync or xtrabackup) ● 24-hour delayed slave with all shards (multi-source, TokuDB)
  • 40. 40© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0 MySQL at Wikipedia Maintenance ● No maintenance windows – code deployments 24/7 ● No integrated system- depending on the change: – pt-online-schema-change/ online schema change – Always enough redundancy for switchover – Batched update https://wikitech.wikimedia.org/wiki/Deployments
  • 41. 41© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0 MySQL at Wikipedia Lessons learned about recovery ● Avoid flopping services: STONITH ● Chaos/monkey testing (we call it deployment schedule) ● Backups are useless: have a faster recovery plan – Data recovery <> service recovery ● Avoid active-passive setups: – Avoid failover -you won't be ready when needed – Have redundancy and a 30% resource utilization ● Automatize and log everything (even if run manually)
  • 42. 42© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0 MySQL at Wikipedia Monitorization ● “Ecosystem” problem: too many of them – Ganglia: basic parameters – Icinga: alerts – Graphite & Graphana: custom graphs – Logstash: centralization of logs ● Application db errors and slow queries – Custom DB monitoring system: “Tendril” ● Graphs, slow queries and reports – pt-query-digest ● Ishmael web interface (deprecated)
  • 43. 43© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0 MySQL at Wikipedia CHALLENGES MySQL at Wikipedia
  • 44. 44© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0 MySQL at Wikipedia Infrastructure and code ● Writes are not an issue for us -reads are – Logged users and POST requests are not cached ● 15 year old PHP application means technical debt – Dependency on statement-based replication – No real utf-8 support at the time – No sql_mode set (WIP)
  • 45. 45© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0 MySQL at Wikipedia Best things about MySQL ● InnoDB is reliable ● Easy to use ● Fast ● Not trying to be smart ● Wide 3rd party support (utilities)
  • 46. 46© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0 MySQL at Wikipedia Worst things about MySQL ● Many manual operations (provisioning, replication, HA, partitioning) – They have to be automated by us – Some of them are slowly being implemented ● Lack of proper compression (both reliable and performant)
  • 47. 47© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0 MySQL at Wikipedia Future (I) ● SSDs and vertical scaling ● Compression (InnoDB, RocksDB, TokuDB?) ● OLAP/Column based solution for analytics ● Fully Active-Active over several datacenters – Multimaster? ● Better maintenance and recovery automation
  • 48. 48© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0 MySQL at Wikipedia Future (II) ● Integrated query analysis and debugging (P_S?) ● Better monitorization – Smoke tests for data integrity, strange states, etc. ● 10.1? 5.7? WebscaleSQL? Galera? ● Better sanitization process (binlog processor) ● Rearchitecture connection handling
  • 49. 49© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0 MySQL at Wikipedia You can help us! ● Apply for the DBA full time position: http://grnh.se/0y4pxm ● Clone our puppet repo and start sending us patches – Or create your own wiki-based tool on Tool-Labs ● Join us at #wikimedia-operations and #wikimedia-databases at Freenode
  • 50. 50© 2015 Wikimedia Foundation & Jaime Crespo. http://wikimediafoundation.org. License: CC-BY-SA-4.0 MySQL at Wikipedia Q&A