MySQL Schema Design in Practice

MySQL Schema design in practice
© 2016 Wikimedia Foundation & Jaime Crespo. http://wikimedia.org. License: CC-BY-SA-3.0
MySQL Schema Design in Practice
Jaime Crespo
Percona Live Amsterdam 2016
-Amsterdam, 3 Oct 2016-
https://wikitech.wikimedia.org/wiki/User:Jcrespo/plam16

2
Agenda
0. Introduction & setup 5. Case #5: Revisions and deletions
1. Case #1: Random pages 6. Case #6: A large table
2. Case #2: Supporting 290
Languages
7. Case #7: What links here
3. Case #3: An abnormal
denormalization
8. Case #8: Anecdotes: The ghost
tables and Timestamps
4. Case #4: Key-value system 9. Case #9: Slots

3
INTRODUCTION & SETUP

4
●
Sr. Database Administrator at
Wikimedia Foundation
●
Used to work as a trainer for
Oracle (MySQL), as a
Consultant (Percona) and as a
Freelance administrator
(DBAHire.com)
This is me fighting bad query
performance

5
Schema design is key for query performance
●
Check my past presentations at:
http://www.slideshare.net/jynus/

6
Mediawiki as the example application
• Mediawiki code distributed under GPL 2 or later
• All Wikimedia project's data licensed under CC-BY-SA-2.5

7
Accessing Wikimedia Production Database (I)
●
Login or register a Wikimedia SUL account (for example,
on https://en.wikipedia.org )
●
Use that account to authenticate on Quarry:
http://quarry.wmflabs.org/
●
Send your queries to the right database (for example,
enwiki_p)!

8
Accessing Wikimedia Production Database (II)

9
Session Dynamic
●
A real database design problem is presented
●
A brief discussion starts (5-10 minutes top)
– If you know the answer, let other people do proposals first;
crazy ideas are encouraged
– assume anything you need
– this is the place to be wrong, not to show of
●
We analyze the proposals, balance its strengths and
weaknesses and compare them with the one in use

10
Case #0: Designing a schema for Wikipedia
●
Which entities would we need?
●
Which relationships?
●
What kind of queries are probably the most common
ones?
●
What do you think are the main scalability and pain
points?

11
Potential entities related to content
●
Page: do we need one?
●
Edit: diferent from page?
●
Dif: Should it be a first-class entity?
●
Revision: Large table?

12
What about page types and properties?
●
Talk pages: Same entity or separate table?
●
Categories, images (files), redirections: are they regular
pages or should they be stored on its own entity?
●
Similar question for image description pages
●
Categories and tags for pages/revisions: how to implement
them?
●
Protection: Some pages can have restrictions on who can edit
them
●
Other properties (tags?)

13
Identifiers
●
How to solve the problem of 2 pages that should have
the same title: Ben Hur (1959 film) and Ben Hur (2016
film)?
●
Should we use the page name or an arbitrary id to
identify a page?
●
What about revision ids?
●
If we needed it, should it be a UUID or a numerical id?

14
Brainstorming time

15
Mediawiki Schema

16
Disclaimers
●
The best solution on paper is not necessarily the best
on production
– It may be too difficult to migrate existing logic (15-
year old application) or not worth it
– Performance is not the only metric: Security,
scalability, reliability, simplicity, etc.
●
You will see many examples of compromises like this on
our application

17
CASE #1: RANDOM PAGES

18
Problem #1 Description
●
At the left of each page there is a link to a
“Random article”
●
It should allow to filter by namespace (e.g.
not all pages are articles)
●
It is a relatively important page compared
to, let’s say, Google’s “I am feeling lucky”,
as it will give an overview of what kind of
articles you will find in a project

19
Problem #1 Restrictions
●
It has to work as fast as a regular page, but
for obvious reasons cannot be cached
●
It has to be realistically pseudorandom
●
It has to always return a result
●
It has to work on a continuously increasing
number of pages, and scale from 1 to
millions

20
Potential solutions
● ORDER BY rand() LIMIT 1
●
Use a well-distributed integer id, use it to get one at
random
●
Questions?
– What indexes would be beneficial on each case?
– How to count the number of total ids?
– How to handle deletions?

21
Brainstorming time

22
Actual solution: Table design (I)
CREATE TABLE /*_*/page (
[…]
-- A page name is broken into a namespace and a title.
-- The namespace keys are UI-language-independent constants,
-- defined in includes/Defines.php
page_namespace int NOT NULL,
-- The rest of the title, as text.
-- Spaces are transformed into underscores in title storage.
page_title varchar(255) binary NOT NULL,
-- 1 indicates the article is a redirect.
page_is_redirect tinyint unsigned NOT NULL default 0,
[…]
-- Random value between 0 and 1, used for Special:Randompage
page_random real unsigned NOT NULL,

23
Actual solution: Table design (II)
[…]
CREATE INDEX /*i*/page_random ON /*_*/page
(page_random);

24
Actual solution: Relevant code
protected function getQueryInfo( $randstr ) {
$redirect = $this->isRedirect() ? 1 : 0;
$tables = [ 'page' ];
$conds = array_merge( [
'page_namespace' => $this->namespaces,
'page_is_redirect' => $redirect,
'page_random >= ' . $randstr
], $this->extra );
$joinConds = [];
// Allow extensions to modify the query
Hooks::run( 'RandomPageQuery', [ &$tables, &$conds, &$joinConds ] );
return [
'tables' => $tables,
'fields' => [ 'page_title', 'page_namespace' ],
'conds' => $conds,
'options' => [
'ORDER BY' => 'page_random',
'LIMIT' => 1,
],
'join_conds' => $joinConds
];
}
From: mediawiki/core/includes/specials/SpecialRandompage.php

25
Actual solution: Query generated
SELECT page_title, page_namespace
FROM page
LEFT JOIN page_props
ON page_id = pp_page AND
pp_propname = ?
WHERE page_namespace IN (…) AND
page_is_redirect = 0 AND
page_random >= $rand
ORDER BY page_random
LIMIT 1;

26
Actual solution: Performance (I)
mysql> EXPLAIN SELECT … G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: page
type: range
possible_keys: name_title,page_random,page_redirect_namespace_len
key: page_random
key_len: 8
ref: NULL
rows: 20473233
Extra: Using where
*************************** 2. row ***************************
id: 1
select_type: SIMPLE
table: page_props
type: eq_ref
possible_keys: PRIMARY,pp_propname_page,pp_propname_sortkey_page
key: PRIMARY
key_len: 66
ref: enwiki.page.page_id,const
rows: 1
Extra: Using where; Using index; Not exists

27
Actual solution: Performance (II)
mysql> SELECT … FROM sys.x$statement_analysis …
*************************** 1. row ***************************
exec_count: 27126203
max_latency: 450802755000
avg_latency: 755698000
lock_latency: 2224515688000000
rows_sent: 27125869
rows_sent_avg: 1
rows_examined: 0
rows_examined_avg: 0
rows_affected: 0
rows_affected_avg: 0
tmp_tables: 0
tmp_disk_tables: 0
rows_sorted: 777598
sort_merge_passes: 0
+------------------------------+
| sys.format_time(avg_latency) |
+------------------------------+
| 755.70 us |
+------------------------------+

28
CASE #2: SUPPORTING 290
LANGUAGES

29
Wikipedia launch and early growth
●
Wikipedia was launched on January 15, 2001, as a single
English-language edition
●
By August 8, 2001, Wikipedia had over 8,000 articles.
●
On September 25, 2001, Wikipedia had over 13,000
articles.
●
By the end of 2001, it had grown to approximately
20,000 articles and 18 language editions.
References: https://en.wikipedia.org/wiki/Wikipedia#History

30
Single-tenancy vs Multi-tenancy
●
1 database per wiki:
– Easier to code
– Easier to scale (?) - you can move wikis to a diferent
server
●
Several wikis on a single database
– More efficiency, specially for small wikis
– They can share existing user database

31
Should we Shard?
●
From early on, people was telling us “you need to shard
to scale”
●
Is it really such a bad idea? When is it needed? When can
it be avoided?
●
If we shard, based on which key?

32
Brainstorming time

33
Wikimedia solution
●
As of this lines, the Wikimedia Foundation hosts 897
wiki-like projects (diferent types and languages)
●
They are divided on 7 “shards” (functional partitions)
– 1 master per shard and datacenter
– Multiples slaves sharing the read load
●
English wikipedia has its own separate shard (s1)
●
s3 host most of the wikis (892)

34
Functional partitioning
Source: https://dbtree.wikimedia.org

35
No sharding
●
Our number of edits is “low” compared to our reads
– Only 500-3000 logical page edits per minute
https://grafana.wikimedia.org/dashboard/db/edit-co
unt
– That means 2000-8000 unique rows written per
second
https://grafana.wikimedia.org/dashboard/db/mysql-
aggregated
– Compared to ~300K total QPS (10-40M rows read/s)

36
Lessons learned: users were separate for years
●
Users were required to register on each project and
language independently
●
Diferent users had the same name registered on
diferent wikis
●
From discussion to universal deployment, it took almost
10 years:
https://meta.wikimedia.org/wiki/Help:Unified_login

37
Unicode

38
Escaping Latin1
●
The mission of the Wikimedia Foundation is to provide
free content in every language
●
That was not possible with Latin1
– It only supports Western languages

39
Multi-language support was limited back in the
day
●
How many of you has ever created a database in
latin1_swedish_ci ?
●
Real UTF-8 support beyond the BMP was added in
MySQL 5.5 (utf8mb4)
●
Still today, latest collation support is relatively new

40
Requirements
●
Full support of all available character sets in the world
●
Support for fully customizable ordering (e.g. entries
within categories), it can be diferent depending on the
language
●
It has to work with available technology 15 years ago

41
Brainstorming time

42
Wikimedia solution: character set
●
Text-like fields are stored in binary fields
– Technically, they are strings with the binary charset
set
●
Latest versions of Mediawiki allow utf8mb4, too
– It wouldn’t work for Wikimedia sites, collation has
been traditionally very limiting

43
tables.sql
CREATE TABLE /*_*/user (
user_id int unsigned NOT NULL PRIMARY KEY
AUTO_INCREMENT,
user_name varchar(255) binary NOT NULL default '',

44
Wikimedia solution: collation
●
List of ordered articles are avoided
●
Whenever custom ordering is needed, an additional,
indexed field is used to allow per-table configurable
ordering
●
Whenever the ordering has to be changed, only row-
level changes have to be done, instead of ALTER TABLEs

45
$wgCategoryCollation
●
https://www.mediawiki.org/wiki/Manual:$wgCategoryCollation
if ( !$dryRun ) {
$dbw->update(
'categorylinks',
[
'cl_sortkey' => $newSortKey,
'cl_sortkey_prefix' => $prefix,
'cl_collation' => $collationName,
'cl_type' => $type,
'cl_timestamp = cl_timestamp',
],
[ 'cl_from' => $row->cl_from, 'cl_to' => $row->cl_to ],
__METHOD__
);
}

46
Unicode is not the only challenge

47
CASE #3: AN ABNORMAL
DENORMALIZATION

48
Users on Wikimedia projects
●
An account is not needed to view the content
●
An account is not needed to edit the content
(anonymous edits)
●
Registered users get some advantages:
– Better tools for editing
– Persistent configurable preferences

49
Content review
●
There must be a way to see the edits from the same
users (all edits must be publicly review-able)
●
There must be a way to block “vandalisms”
(misbehaving users, both registered and unregistered)
●
In extreme cases, there must be a way to protect
content from certain user groups (page protections)
●
Only trusted users should be able to destroy content

50
How to represent users?
●
Should we use strings or arbitrary numerical ids?
– If we use strings, how to rename registered users?
– If we use arbitrary ids for registered users, how to
reference non-registered ones?
●
How to be able to block returning users, including
anonymous ones?
●
How to allow anonymous edits on countries with
doubtful privacy laws?

51
Brainstorming time

52
Wikimedia solution
●
Registered users have a local user id and a string
identifier
●
Local wiki accounts are linked to a global unified
account (SUL) on “centralauth” database
●
Anonymous users are identified by its IPv4 or IPv6
string
●
In general, editions store both the id and the user text

53
Table revision
CREATE TABLE `revision` (
`rev_id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`rev_page` int(10) unsigned NOT NULL,
`rev_text_id` int(10) unsigned NOT NULL,
`rev_comment` tinyblob NOT NULL,
`rev_user` int(10) unsigned NOT NULL DEFAULT '0',
`rev_user_text` varbinary(255) NOT NULL DEFAULT '',
[…]
) ENGINE=InnoDB AUTO_INCREMENT=N DEFAULT
CHARSET=binary

54
Actual user content (registered user)
mysql> SELECT * FROM revision WHERE rev_id = 742218408G
************************ 1. row ************************
rev_id: 742218408
rev_page: 46812822
rev_text_id: 750242058
rev_comment: Testing revision comment
rev_user: 25118340
rev_user_text: JCrespo (WMF)
rev_timestamp: 20161002111252
...

55
Actual user content (anonymous user)
mysql> SELECT * FROM revision WHERE rev_id = 742219056G
************************ 1. row ************************
rev_id: 742219056
rev_page: 46812822
rev_text_id: 750242734
rev_comment: As an anonymous user, my public IP
will get saved, instead of a username
rev_user: 0
rev_user_text: 80.113.15.100
rev_timestamp: 20161002111851
...

56
Pros and cons of the current implementation
●
By denormalizing the table, most of the time only this
table has to be checked
– Only when a user is clicked the user table is accessed
– No need to store information for the large amount
of anonymous users with very few edits
●
User renames are painful database-wise, and almost
impossible for users with huge amount of edits (like
bots)

57
CASE #4: KEY-VALUE SYSTEM

58
Content storage
●
In the typical mediawiki installation content is
referenced this way:
– Pages can have several revisions, by default the last
one is parsed and displayed
– Revisions point to a text row
– Text contains wikitext that has to be parsed
“rendered” and sent to the user

59
Wikimedia sites needs to return
hundreds thousands of pages per second
●
The size of the content (wikitext) is multiple times that of the
metadata
●
The database growth is also diferent from the metadata, and
very diferent for each wiki
●
Should we setup a separate key-value system to store those
edits? What should we seek?
– Compression
– Automatic sharding
– Automatic failover
– JSON support for flexible datatypes

60
Brainstorming time

61
Wikimedia solution
●
Each page can have a diferent content model/storage
●
For example:
– Regular Wikitext pages
– User-editable JS/CSS/application messages
– Forum threads (Flow feature, etc.)
– Any other created by new extensions

62
Wikitext storage
●
The “text” table only contains pointers to content, not real content
mysql> SELECT * FROM text ...;
+---------+-----------------------+---------------------+
| old_id | old_text | old_flags |
+---------+-----------------------+---------------------+
| 1 | #REDIRECT [[Town of 1770]] | utf-8 |
| 2 | #REDIRECT [[Project:One-liner listings]] | utf-8 |
...
| 3027206 | DB://cluster24/545108 | utf-8,gzip,external |
| 3027205 | DB://cluster25/544444 | utf-8,gzip,external |
+---------+-----------------------+---------------------+

63
External Storage
●
Several shards of MySQL servers can serve content for every
wiki
●
The text is compressed and decompressed in gzip format at
application level
●
Smart “compressing” can be done:
– Reviews with the same content for the same page are
deduplicated
– Older revisions are stored only using difs

64
External Storage Cluster

65
Solid performance over HDs
mysql> SELECT
digest, digest_text, sum(COUNT_STAR),
sum(SUM_TIMER_WAIT)/SUM(COUNT_STAR)/1000000 as microseconds,
min(first_seen), max(last_seen)
FROM events_statements_summary_by_digest
GROUP BY DIGEST ORDER BY sum(COUNT_STAR) DESC LIMIT 1G
*************************** 1. row ***************************
digest: 1bf861e8cd3ea6bcac323bdf9caf4876
digest_text: SELECT `blob_text` FROM `blobs_cluster25` WHERE `blob_id`
= ? LIMIT ?
sum(COUNT_STAR): 2356927126
microseconds: 3384.60227946
min(first_seen): 2015-12-09 15:00:45
max(last_seen): 2016-10-02 11:41:01

66
Speed optimizations
●
External storage is only the canonical place where data is stored
– Several layers of caching makes them low-traffic, disk-based
storage
●
Parsercache in memory (memcached) and disk (parsercache
mysqls) avoids frequent usage
– Local memcache is fast
– Parsercache is shared even between datacenters, and
survives restarts

67
CASE #5: REVISIONS AND
DELETIONS

68
Deletions
●
Pages can be deleted, with several degrees:
– A new revision could just override a previous version of the page
– A deletion that could be restored afterwards
– Personal information or copyright that must be hidden from
everyone
●
In some cases, only some revisions should be deleted, not the whole
page- e.g. someone editing an otherwise legitimate page with
someone’s private data

69
How to implement deletions and
restores?
●
Should we delete content rows when doing hard-
deletes or just overwrite the them with garbage?
●
Should we move the rows to an archive table or should
we mark them with deleted=1?

70
Brainstorming time

71
Wikimedia implementation
●
Individual revisions can be hidden (“suppresed”), no matter
the page status
●
Normal procedure is that full pages have to be deleted:
– In that case, revisions are moved to the archive table
– The page entry is deleted
●
On restore, revisions are moved back to the revision table
– A new page, with a new page_id is created
– Not all revisions have to be restored necessarily

72

73
Deleting a page Code
/**
* Back-end article deletion
* Deletes the article with database consistency, writes logs, purges caches
…
*/
public function doDeleteArticleReal(
$reason, $suppress = false, $u1 = null, $u2 = null, &$error = '', User $user = null,
$tags = []
) {
$dbw = wfGetDB( DB_MASTER );
$res = $dbw->select(
'revision',
array_merge( $fields, $deletionFields ),
[ 'rev_page' => $id ],
__METHOD__,
'FOR UPDATE'
);
$dbw->insert( 'archive', $rowsInsert, __METHOD__ );
// Now that it's safely backed up, delete it
$dbw->delete( 'page', [ 'page_id' => $id ], __METHOD__ );
$dbw->delete( 'revision', [ 'rev_page' => $id ], __METHOD__ );
// Log the deletion, if the page was suppressed, put it in the suppression log instead
$logEntry = new ManualLogEntry( $logtype, 'delete' );
INSERT … SELECTs are
avoided on HEAD

74
Lessons learned
●
INSERT … SELECTs are painful both for performance
and/or consistency reasons
– They created actual issues when combined with
filtering or on pages with many revisions
– New implementation avoids them
●
Moving rows between tables is a terrible idea
– Specifically, our implementation makes almost
impossible to track the history of a text

75
CASE #6: A LARGE TABLE

76
Revision table on enwiki
MariaDB MARIADB s1-master enwiki > SHOW CREATE TABLE revision;
CREATE TABLE `revision` (
`rev_id` int(8) unsigned NOT NULL AUTO_INCREMENT,
…
) ENGINE=InnoDB AUTO_INCREMENT=741078657 DEFAULT CHARSET=binary
MariaDB MARIADB s1-master enwiki > SHOW TABLE STATUS like 'revision'G
*************************** 1. row ***************************
Name: revision
Engine: InnoDB
Version: 10
Row_format: Compact
Rows: 614837124
Avg_row_length: 153
Data_length: 94510252032
Max_data_length: 0
Index_length: 90206896128

77
Point SELECTs are fast
MariaDB PRODUCTION s1 localhost performance_schema > SELECT * FROM
events_statements_summary_by_digest ORDER BY count_star DESC LIMIT 1G
*************************** 1. row ***************************
SCHEMA_NAME: enwiki
DIGEST: ed3c3539910af27e0f7e4ea442db9124
DIGEST_TEXT: SELECT `page_id` , `page_len` ,
`page_is_redirect` , `page_latest` , `page_content_model` FROM `page` WHERE
`page_namespace` = ? AND `page_title` = ? LIMIT ?
COUNT_STAR: 37247176536
SUM_TIMER_WAIT: 6633094681562378000
MIN_TIMER_WAIT: 43209000
AVG_TIMER_WAIT: 178083000
MAX_TIMER_WAIT: 258818851000
SUM_LOCK_TIME: 1867944737116000000
...
1 row in set (0.06 sec)
+---------------------------------+
| sys.format_time(AVG_TIMER_WAIT) |
+---------------------------------+
| 178.08 us |
+---------------------------------+

78
Ranges are slow
Host User Schema Client Source Thread Transaction Runtime Stamp
db1066 wikiuser enwiki mw1193 - 42910008229 133321827778 318s 2016-09-25
15:31:15
SELECT /* ApiQueryContributors::execute */ rev_page AS `page`, rev_user AS `user`,
MAX(rev_user_text) AS `username` FROM `revision` WHERE rev_page = '6768170' AND
(rev_user != 0) AND ((rev_deleted & 4) = 0) GROUP BY rev_page, rev_user ORDER BY
rev_user LIMIT 501

79
revision is not the only tall table
MariaDB PRODUCTION s1 localhost enwiki > SHOW CREATE TABLE loggingG
*************************** 1. row ***************************
Table: logging
Create Table: CREATE TABLE `logging` (
`log_id` int(10) unsigned NOT NULL AUTO_INCREMENT,
...
) ENGINE=InnoDB AUTO_INCREMENT=77577756 DEFAULT CHARSET=binary
MariaDB PRODUCTION s1 localhost enwiki > SHOW TABLE STATUS like 'logging'G
*************************** 1. row ***************************
Name: logging
Engine: InnoDB
Version: 10
Row_format: Compact
Rows: 72871150
Avg_row_length: 164
Data_length: 11963203584
Max_data_length: 0
Index_length: 36273225728

80
Brainstorming time

81
recentchanges table
-- Primarily a summary table for Special:Recentchanges,
-- this table contains some additional info on edits from
-- the last few days, see Article::editUpdates()
–
CREATE TABLE /*_*/recentchanges (
rc_id int NOT NULL PRIMARY KEY AUTO_INCREMENT,
rc_timestamp varbinary(14) NOT NULL default '',
-- As in revision
rc_user int unsigned NOT NULL default 0,
rc_user_text varchar(255) binary NOT NULL,
More on:
https://phabricator.wikimedia.org/difusion/MW/browse/master/maintenance/tables.sql;bc05426ae2708f8ac23b9106911fe35b5c51fd30$1057

82
recentchanges usage
●
Recentchanges is mainly a revision summary table
– Most reviews are only of recent edits -entries are
purged after 30 days:
MariaDB MARIADB s1-master enwiki > SELECT now(), min(rc_timestamp) from recentchanges;
+---------------------+-------------------+
| now() | min(rc_timestamp) |
+---------------------+-------------------+
| 2016-09-25 09:04:35 | 20160826090422 |
+---------------------+-------------------+
●
It is updated synchronously with the edits
●
It also contains additional fields / related tables like tags

83
Special slaves && partitioning
●
Mediawiki allows to define instance groups for certain
queries https://noc.wikimedia.org/conf/highlight.php?file=db-eqiad.php
'db1051' => 50, # 2.8TB 96GB, watchlist, recentchanges, contributions, logpager
'db1055' => 50, # 2.8TB 96GB, watchlist, recentchanges, contributions, logpager
●
Querying contributions uses a separate group
contributions:
https://phabricator.wikimedia.org/difusion/MW/browse/master/includes/specials/pagers/ContribsPager.php;bc05426ae2708f8ac23b9106911fe35b5c51fd30$73
// Most of this code will use the 'contributions' group DB, which can map to replica Dbs
// with extra user based indexes or partioning by user. The additional metadata
// queries should use a regular replica DB since the lookup pattern is not all by user.
$this->mDbSecondary = wfGetDB( DB_REPLICA ); // any random replica DB
$this->mDb = wfGetDB( DB_REPLICA, 'contributions' );

84
enwiki tables have special partitioning
ALTER TABLE enwiki.revision
DROP PRIMARY KEY,
DROP INDEX rev_id,
ADD PRIMARY KEY (rev_id, rev_user)
PARTITION BY RANGE (rev_user) (
PARTITION p1 VALUES LESS THAN (1),
…
More on: https://phabricator.wikimedia.org/diffusion/OSOF/browse/master/dbtools/s1-pager.sql

85
The solution has problems
●
It is a server-side hack, with poor to no awareness on
server side
– Mediawiki has to support MySQL 5.0, which has no
patitioning support
●
It is an enwiki-only patch; which is a bad idea in general
●
Special slaves are a threat to High Availability

86
CASE #7: WHAT LINKS HERE

87
Categories, templates, images and special
pages
●
There is a lot of information that needs to be updated
when a new edits is done:
– The category must include the new page, on its right
position
– If it is a template or an image, all pages that include it
have to change
– The “What links here” has to reflect the new links
– If a new page is created, links to it have to go from red
to blue color

88
Latency on edit has to be low
●
Users understand that and “edit” will be slower than
loading a page, but it still cannot take more than 0.5-1
second tops:
https://grafana.wikimedia.org/dashboard/db/save-timing
●
But we just said that an edit may require an update on
millions of others!
●
How to implement all the previous changes?

89
Brainstorming time

90
Wikimedia solution
●
Only immediate tasks are done synchronously,
transactionally:
– Wikitext content is saved on External Storage
– The recentchanges table adds a new entry for
reviewing
– The user edit count is increased in some wikis
– The page is parsed for display to the user

91
Background jobs
●
Many tasks are enqueued on a Redis job queue
●
Most of those tasks are cached/denormalized on database
tables for easy joining:
– Add the page to the proper category (categorylinks)
– Add the page to the proper links (pagelinks)
– Add the page to the list of templates used (templatelinks)
– Many others such as refreshing the list of titles and its
index on elasticsearch

92
doRefreshLinks() Job
function run() {
// Job to update all (or a range of) backlink pages for a page
$this->runForTitle( $this->title );
}
protected function runForTitle( Title $title ) {
$revision = Revision::newFromTitle( $title, false, Revision::READ_LATEST );
$parserOutput = $content->getParserOutput( $title, $revision->$title, $revision-
>$title, $revision getId(), $parserOptions, false );→
$updates = $content->getSecondaryDataUpdates($title, null, !empty( $this, $revision-
>$title, $revision->getId(), $parserOptions, false );
$updates = $content getSecondaryDataUpdates(→
$title,
null,
!empty( $this->params['useRecursiveLinksUpdate'] ),
$parserOutput
);
InfoAction::invalidateCache( $title );

93
*links tables
●
They store the page_ids where the resource is used but
the namespace and the title they include, not the ids
●
This is because pages reference a title, not an entity
(e.g. you can include the template {{stub}}, or link to a
page that can or not exist
●
Some of those tables can grow a lot, and store very
redundant data (there could be millions of rows with
the template {{cc-by-sa-2.5}} on Wikimedia Commons

94
Example: PageLinks Table
SHOW CREATE TABLE pagelinksG
*************************** 1. row ***************************
Table: pagelinks
Create Table: CREATE TABLE `pagelinks` (
`pl_from` int(8) unsigned NOT NULL DEFAULT '0',
`pl_namespace` int(11) NOT NULL DEFAULT '0',
`pl_title` varbinary(255) NOT NULL DEFAULT '',
`pl_from_namespace` int(11) NOT NULL DEFAULT '0',
UNIQUE KEY `pl_from` (`pl_from`,`pl_namespace`,`pl_title`),
KEY `pl_namespace` (`pl_namespace`,`pl_title`,`pl_from`),
KEY `pl_backlinks_namespace`
(`pl_from_namespace`,`pl_namespace`,`pl_title`,`pl_from`)
) ENGINE=InnoDB DEFAULT CHARSET=binary

95
Future
●
A lot of space could be saved by normalizing/deduplicating the title
text:
mysql> SHOW TABLE STATUS like '%links';
+--------------------+--------+---------+------------+------------+----------------+-------------+-----------------+--------------+
| Name | Engine | Version | Row_format | Rows | Avg_row_length | Data_length | Max_data_length | Index_length |
+--------------------+--------+---------+------------+------------+----------------+-------------+-----------------+--------------+
| categorylinks | InnoDB | 10 | Compact | 106308892 | 210 | 22379020288 | 0 | 30838489088 |
| externallinks | InnoDB | 10 | Compact | 91491185 | 279 | 25571622912 | 0 | 40650997760 |
| imagelinks | InnoDB | 10 | Compact | 81270504 | 83 | 6765756416 | 0 | 9644818432 |
| iwlinks | InnoDB | 10 | Compact | 16909955 | 95 | 1622573056 | 0 | 2367488000 |
| langlinks | InnoDB | 10 | Compact | 26155404 | 74 | 1941733376 | 0 | 1434042368 |
| msg_resource_links | InnoDB | 10 | Compact | 3524 | 130 | 458752 | 0 | 0 |
| pagelinks | InnoDB | 10 | Compact | 1192043172 | 74 | 89018269696 | 0 | 122491224064 |
| templatelinks | InnoDB | 10 | Compact | 616611334 | 80 | 49860050944 | 0 | 66098577408 |
+--------------------+--------+---------+------------+------------+----------------+-------------+-----------------+--------------+
●
That would make title a first-class entity, diferent from the page
entity

96
CASE #8: ANECDOTES: THE
GHOST TABLES AND
TIMESTAMPS

97
Story time
●
A recently pooled slave broke its replication with:
Error 'Table 'idwiki.hitcounter' doesn't exist' on
query. Default database: 'idwiki'. Query: 'DELETE
FROM `idwiki`.`hitcounter`'
●
The table was indeed non-existent, but it was
deprecated and was unused by mediawiki
●
Hackers? Replication bug? Someone doing maintenance
out-of-band?

98
Brainstorming time

99
Architecture
●
At the moment, Wikimedia servers mostly use
STATEMENT-based replication due to several
application dependencies
●
A Master-Master active/passive replication was being
used among datacenters
●
Replication was temporarily routed through a slave to
deploy new TLS certificates
●
The middle slave was rebooted

100
hitcounter table was using the MEMORY engine
●
On server restart, because the server to avoid
replication issues, a DELETE command is sent
●
The DELETE got replicated towards the remote master,
back to the primary master, and finally to the new slave
– This broke replication because the new slave didn’t
have the obsolete table, despite not writing to it
●
Remember to clean up your tables on all servers to
avoid issues like this!

101
Timestamps and MySQL < 4.1
●
There where automatically updated on INSERT and
UPDATE with no possibility of disabling that
●
No strict mode disallowing zero dates
●
Diferent databases and standards support needed:
https://www.mediawiki.org/wiki/Manual:WfTimestamp
#Formats

102
How to implement timestamps?
●
They must store UTC times, controlled at application
side
●
They must be strictly sortable to be used on listings
●
They must work on mysql 4.0
●
They must work similarly for other database backends

103
Brainstorming time

104
Mediawiki solution
●
Timestamps are stored as binary(14)
https://www.mediawiki.org/wiki/Manual:Timestamp
●
For backwards and join compatibility, that is still the
preferred format
●
TIMESTAMP (4 bytes) now has all required features and
would be much more compact, but converting millions
of records is not worth the efort right now

105
tables.sql
-- The MySQL table backend for MediaWiki currently uses
-- 14-character BINARY or VARBINARY fields to store timestamps.
-- The format is YYYYMMDDHHMMSS, which is derived from the
-- text format of MySQL's TIMESTAMP fields.
–
-- Historically TIMESTAMP fields were used, but abandoned
-- in early 2002 after a lot of trouble with the fields
-- auto-updating.
–
-- The Postgres backend uses TIMESTAMPTZ fields for timestamps,
-- and we will migrate the MySQL definitions at some point as
-- well.
CREATE TABLE /*_*/logging (
-- Log ID, for referring to this specific log entry, probably for deletion and such.
log_id int unsigned NOT NULL PRIMARY KEY AUTO_INCREMENT,
-- Symbolic keys for the general log type and the action type
-- within the log. The output format will be controlled by the
-- action field, but only the type controls categorization.
log_type varbinary(32) NOT NULL default '',
log_action varbinary(32) NOT NULL default '',
-- Timestamp. Duh.
log_timestamp binary(14) NOT NULL default '19700101000000',

106
CASE #9: SLOTS

107
Structured data for Wikipedia
●
Wikitext is powerful but complex
– It is not easy to do changes such as categories,
infoboxes (templates)
– It is also not easy to ofer them in a computer-
readable way
●
Tools like wikidata or image metadata require an easier
way to integrate structured data

108
How to implement Multi-Content Revisions?
●
A page may combine multiple sources (Image & Image
metadata, wikitext and discussions, etc.)
●
A new revision could be created by editing regular text
or some of the structured data
●
They types of structured data may vary and could be
added later with new extensions

109
Brainstorming time

110
Current Multi-Content Revision proposal
●
page and revision will work as usual (except a few of its
fields will be made redundant)
●
2 new tables: content and content_revision will be
added
●
content initially will have the metadata for wikitext
●
Other types of content can be referenced by revision
thought content_revision
– A revision now can handle multiple contents

111
The idea is to multiplex revision
●
https://phabricator.wikimedia.org/T107595
●
We go from the straight forward
– page -> revision -> text ( -> external store )
●
To a more indirect model:
– page -> ( revision -> ) slots -> content ->
( text | external store )

112
The proposal is not issue-free
●
Revision table is already very large for non-point SELECTs
– How to handle ranges when content will be an even taller
table (several content types per revision)
– Also previously independent tables will now be
integrated there (image_revisions, etc.)
●
Maybe a diferent, multi-table implementation for the
polymorphic association would be preferred?
– Normalization and easiness to code vs. performance on
implementation for large wikis

113
FINAL REMARKS

114
“Do not let reality spoil your perfect on-paper
design”
●
Not always the most elegant design is the most
appropriate
– Reliability is the enemy of performance: simple, fast,
safe – chose two
– Technology available now does not normally cover
100% of the use cases; chose the one that cover 99%,
implement the other 1%

115
Where to know more about design
●
Learn how other are doing it:
– Uber:
●
https://eng.uber.com/schemaless-part-one/
●
https://www.percona.com/live/plam16/sessions/relational-databases-uber-mysql-postgres
– Facebook:
●
https://www.percona.com/live/plam16/sessions/massive-schema-changes-facebook
●
https://www.facebook.com/MySQLatFacebook/
– Youtube:
https://www.percona.com/live/plam16/sessions/launching-vitess-how-run-youtubes-mysql-sharding-engine
– Pinterest: https://engineering.pinterest.com/blog/sharding-pinterest-how-we-scaled-our-mysql-fleet
●
Chose the right literature:
– https://pragprog.com/book/bksqla/sql-antipatterns

116
Where to know more about Mediawiki /
MySQL@Wikipedia
●
MySQL at Wikipedia Introduction:
https://www.mediawiki.org/wiki/File:MySQL
_at_Wikipedia.pdf
●
Mediawiki source code and documentation:
https://www.mediawiki.org/wiki/MediaWiki
●
Wikitech (technical documentation):
https://wikitech.wikimedia.org/
●
Operations/puppet (infrastructure) git repository:
https://phabricator.wikimedia.org/difusion/OPUP/

117
Was this session helpful?
●
Consider supporting the Wikimedia Foundation!
– The Wikimedia Foundation, Inc. is a nonprofit charitable organization
dedicated to encouraging the growth, development and distribution of
free, multilingual, educational content, and to providing the full content
of these wiki-based projects to the public free of charge.
– https://wikimediafoundation.org
●
You can contribute with:
– Your code (including infrastructure!):
https://phabricator.wikimedia.org/difusion/
– Your time: https://en.wikipedia.org/wiki/Help:Editing
– Your money: https://donate.wikimedia.org/

118
Thank You! Remember to rate my session

MySQL Schema Design in Practice

More Related Content

What's hot

Viewers also liked

Similar to MySQL Schema Design in Practice

More from Jaime Crespo

Recently uploaded

MySQL Schema Design in Practice