Mysql Performance Optimization Indexing Algorithms and Data Structures
MySQL Performance Optimization Part II“Indexing Data Structures and Algorithms” Abhijit Mondal Software Engineer at HolidayIQ
ContentsHash IndexesB-Trees and B+ Trees IndexesIndexing Strategies for High PerformanceFull Text Searching
Hash Indexes● A hash index is built on a hash table and is useful only for exact lookups that use every column in the index. For each row, the storage engine computes a hash code of the indexed columns, which is a small value that will probably differ from the hash codes computed for other rows with different key values. It stores the hash codes in the index and stores a pointer to each row in a hash table.● CREATE TABLE user_info (user_id int not null primary key auto_increment, username varchar(50), password char(32), KEY USING HASH(username, password)) ENGINE=MEMORY;● Suppose the has function is f() i.e. f : (username, password) -> Integer, then our data will have has values as such for eg. f(john,abc123) = 2789. The indexs data structure will have a pointer from slot 2789 to the row which has username john and password abc123.● If the function f() is very selective i.e. For each combination of username and password it gives a different integer as output, then lookups will be O(1) in constant time (very very fast). For queries such as SELECT * from user_info where username=john and password=abc123, it will not scan the table but compute f(john,abc123)=2789 and directly pick up the row from slot 2789.
Hash Indexes● ORDER BY queries on Memory engine will not take advantage of hash indexes as rows are not stored in sorted order.● Queries such as SELECT * from user_info where username=john; will not use hash index because to compute the function f() it needs both username and password.● Range queries doesnt use hash indexes because to compute f() it needs exact values for the parameters.● If the function f() is not selective, i.e. For more than one combination of username, password pair it returns the same integer output e.g. f(john,abc123)=2789 and f(mary,25qwer)=2789 and so on for 5 other pairs then the slot 2789 points to a linked list of row pointers where each row pointer in the linked list has username, password pair that gives the same output when f() id applied on it. This case is termed chaining.● In case of hash collisions the worst case perormance for a query like SELECT * from user_info where username=mary and password=25quer; can amount to equivalent of a full table scan if all username, password pairs in the table have the same hash value.
Hash Indexes● Analysis of hashing with chaining : 1. How long does it take to return the output of the query SELECT * from user_info where username=johnny and password=derp123 ? 2. Assuming simple uniform hashing, if there are m slots in the index and a total of n rows then the expected number of rows each slot points to is a=n/m (the average length of linked list for each slot is n/m ). 3. For query such as SELECT * from user_info where username=johnny and password=derp123 the average number of lookups is Θ(1+a). Proof : Suppose the username-password combination we are searching is non- existent then Mysql would compute f(johnny,derp123) = x, then it will search in the linked list of pointers in slot x. Since it is not there it has to search till the end of linked list i.e. Average length of linked list = a = Θ(1+a). If the particular username-password combination is present then the number of lookups is equal to 1+ #(row pointers before (johnny,derp123) in the linked list). For large values of n (number of rows in the table) we can assume that the expected number of row pointers before (johnny,derp123) in its linked list is a/2. Thus average number of lookups = 1+a/2 = Θ(1+a).
Hash Indexes● Hash Indexes for InnoDB engine : The InnoDB storage engine has a special feature called adaptive hash indexes. When InnoDB notices that some index values are being accessed very frequently, it builds a hash index for them in memory on top of B-Tree indexes.● A Good Hash function f() : Each row is equally likely to hash to any of the m slots independently of where any other row has hashed to.i.e. f(john,abc123) should be independent of f(johnny,derp123).● In InnoDB there is no inbuilt hash function that we can take advantage of for “explicit” indexing. So we can maintain one column in the table for our hash values. ALTER TABLE user_info add column hash char(32) key. Then index hash.● Collision analysis using 16 byte (32 hexadecimal digits) MD5() hash function : 1. MD5() hash lookups are time consuming as the algorithm takes time to compute the value and then since the value is 32 digit hexadecimal string comparison also takes time. 2. SELECT * from user_info where username=johnny and password=derp123 and hash=690cdca9655043e9d087a1d50cd74e02; we need the check on username and password field also so that single row is returned in case of collisions.
Hash Indexes● Method 2 : Using CRC32() as another builtin hash function is a better choice than MD5() since it results in a 10 digit integer value which can speed up comparisons effectively. SELECT * from user_info where username=johnny and password=derp123 and hash=3682452828;● Method 3 : Using column prefixes as hash index. We can use fixed length prefixes from our username and password values. For e.g. For username johnny and password derp123 we can choose our hash to be (4+3) character long johnder. 1. SELECT * from user_info where username=johnny and password=derp123 and hash=johnder; 2. Less comparison overhead compared to indexing the whole username and password values. 3. Less selectivity. Defining selectivity s1= (# of distinct username-password pairs)/ (# of rows in user_info) and s2=(# of distinct hash values)/(# of rows in user_info). Choose a length L for our hash values for which s2 ≈ s1, then number of collisions will be minimized.
Hash Indexes● Method 4 : Using universal class of hash functions. Convert our username and password strings to integer by summing up their ASCII character values and assuming the following for them : 1. The ASCII character values for username and passwords lie between 0 and 255. 2. Maximum length of username is 10 and password is 10. Thus the maximum integer value for username is 255*10 and password is 255*10 adding them gives the maximum integer value for our key = 5100. 3. Assuming there are 1000 distinct username passwords in our database, choose a prime p > 5100, p=5101, choose 2 integers 1<= a <= p-1 and 0<= b <= p- 1, let a=19 and b=21.● Let the sum of the ASCII values of username and password be k. Then our universal hash function becomes f(k) = ((ak+b) mod p) mod m, where p=5101, m= number of distinct username-password pairs (1000 in our case), a=19 and b=21. So f(k)= ((19k+21) mod 5101) mod 1000.● For username johnny and password derp123, k = 106 + 111 + 104 + 110 + 110 + 121 + 100 + 101 + 114 + 112 + 49 + 50 + 51 = 1239. Thus f(1239) = 158. Thus our hash value for (johnny,derp123) is 158.
Hash Indexes● Using universal class of hash functions the probability that Pr(f(k)=f(l), k≠l) <= 1/m. Hence in our case probability that f(k)=f(l) is less than 1/1000 = 0.001.● Proof: Let r = (ak+b) mod p and s = (al+b) mod p, then r-s = a(k-l) mod p. But 1<= a < p and (k-l) < p and p is prime hence r≠s (mod p). Since there are p(p-1) pairs for (a,b) and since r≠s (mod p) thus there are p(p-1) pairs for (r,s), there is one- to-one correspondence between (a,b) and (r,s). Thus if collision occurs it is due to for some r = s (mod m). For a given value of 0<= s < p, and r≠s , the number of values for which r = s (mod m) is at most (p-1)/m. Thus the probability that for a particular value of s , r = s (mod m) is at most ((p-1)/m)/(p-1) = 1/m.● Thus programmatically computing f(k) for lookups and the using query : SELECT * from user_info where username=johnny and password=derp123 and hash=158; has great performance benefits.
B-Tree Indexes● B-trees are balanced search trees: height = O log(n) for the worst case.● They were designed to work well on Direct Access secondary storage devices (magnetic disks).●● B-trees (and variants like B+ and B* trees ) are widely used in database systems.
B-Tree Indexes● A B-tree T is a rooted tree (with root root[T]) with properties: Every node x has four fields: 1. The number of keys currently stored in node x, n[x]. 2. The n[x] keys themselves, stored in nondecreasing order: key1[x] ≤ key2[x] ≤ · · · ≤ keyn[x][x] . 3. leaf[x] = “True” if x is a leaf else “False” 4. n[x] + 1 pointers, c1[x], c2[x], . . . , cn[x]+1[x] to its children.● The keys keyi[x] separate the ranges of keys stored in each subtree: if k i is any key stored in the subtree with root ci[x], then: k1 ≤ key1[x] ≤ k2 ≤ key2[x] ≤ . . . ≤ keyn[x] ≤ kn[x]+1 .
B-Tree Indexes● All leaves have the same height, which is the tree’s height h.● There are upper on lower bounds on the number of keys on a node. To specify these bounds we use a fixed integer t ≥ 2, the minimum degree of the B-tree: lower bound: every node other than root must have at least t − 1 keys i.e. At least t children. upper bound: every node can contain at most 2t − 1 keys i.e. every internal node has at most 2t children.●
B-Tree Indexes● SELECT * from user_info where firstname=johnny and lastname=derp and dob=1981-08-14; (InnoDB engine; index on (firstname,lastname,dob));● Search Algorithm : (x : node pointer to some node in a subtree) BTree-MySQL-Search (x=null, firstname=, lastname=, dob=) i=1; while ( i < n[x] and (firstname,lastname,dob) > keyi[x] ) i = i+1; if ( i ≤ n[x] and (firstname,lastname,dob) > key i[x] ) then return keyi[x] -> rows; else if ( leaf[x] ) then return null; else Disk-Read(ci[x]); return BTree-MySQL-Search(ci[x], firstname, lastname, dob );● Number of disk pages accessed by BTree-MySQL-Search Θ(h) = Θ(log t n) where n is the number of rows in the index.
B-Tree Indexes● INSERT, DELETE and UPDATE queries are much more involved. Lets discuss in brief about only INSERT.● INSERT into user_info (firstname,lastname,dob) values (johnny, derp, 1981-08- 14);● Insert algorithm : 1. Lets assume k= (firstname,lastname,dob). If we find the leaf node x where k will be inserted. a. If x is not full, then insert k into x at an appropriate position (in ascending order of keys ). b. If x is full then compute the median value of all the keys in x . Then split the node into 2 nodes about the median. Then k is inserted into one of the splitted nodes at an appropriate position. The median value is then considered inserting into the parent node of x and this process is followed recursively. Moving up the tree if we find that the current root node needs to be split then the root node is split into 2 and our new root node is a single key node with the median value from last split.
B-Tree Indexes● B-Tree insertion demonstration :● The key is always inserted in a leaf node● Requires O(h) = O(logt n) disk accesses.
B-Tree Indexes● B+ Trees are B-Trees with the modification that all internal nodes store the keys that are used in the indexing while the leaf nodes contains both the keys and the rows corresponding to the key.● Types of queries that can use a B-Tree index : 1. Match the full value – SELECT * from user_info where firstname=johnny and lastname=derp and dob=1981-08-14; 2. Match a leftmost prefix – SELECT * from user_info where firstname=johnny; 3. Match a column prefix - SELECT * from user_info where firstname like john%; 4. Match a range of values - SELECT * from user_info where firstname between john and johnny; 5. Match one part exactly and match a range on another part – SELECT * from user_info where firstname=johnny and lastname like de%; 6. InnoDB uses B+Tree indexes, so to take advantage of index-only-queries where rows are returned directly from index, select columns which are indexed - SELECT firstname, lastname from user_info where firstname like john%;
B-Tree Indexes● Types of queries that cant use a B-Tree index : 1. They are not useful if the lookup does not start from the leftmost side of the indexed columns – SELECT * from user_info where lastname=derp; SELECT * from user_info where firstname like %p; 2. You can’t skip columns in the index – SELECT * from user_info where firstname=johnny and dob=1981-08-14; 3. The storage engine can’t optimize accesses with any columns to the right of the first range condition – SELECT * from user_info where firstname="johnny" and lastname like de% and dob=1981-08-14;
Indexing Strategies for High Performance● Isolating the Column : “Isolating” the column means it should not be part of an expression or be inside a function in the query. SELECT * from user_info where user_id + 1 = 5; or SELECT * from user_info where TO_DAYS(CURRENT_DATE) - TO_DAYS(dob) <= 365; dont use indexes with MySQL.● Prefix Indexes and Index Selectivity : For BLOB and TEXT columns instead of indexing a very long string , alternative is to index a prefix of the string . But index selectivity is also be taken care of . Index selectivity is the ratio of the distinct number of rows (grouped by our indexed field) to the total number of rows. The prefix length depends on index selectivity. For e.g. If there are 1000 rows in our user_info table and based on city there are 435 distinct rows grouped by city, then our selectivity is 435/1000 = 0.435, now assuming that we choose a prefix length of 3, then the number of distinct rows grouped by city becomes 879 since there are many cities that have same prefix. Increasing the prefix length will always improve selectivity but choosing an optimal value (selectivity closest to 0.435 but length not too high) is important. In our case a prefix length of 7 gives number of distinct rows grouped by city 450. Thus we choose 7 as prefix length. ALTER TABLE user_info ADD KEY (city(7));
Indexing Strategies for High Performance● Choosing a good column order (For multicolumn indexes) : 1. If ORDER BY or GROUP BY is not required then index the columns from left to right in order of selectivity. i.e. The most selective column should be the leftmost so that probability of filtering maximizes for the leftmost column. For e.g the indexing order for the columns country and city should be (city, country) because more users belong to the same country compared to the same city. i.e. Selectivity of city is more than country thus filtering on “where city=kolkata and country =india ” is efficient than “where country=india and city =kolkata ” . 2. In case of ORDER BY or GROUP BY the ORDER BY columns should be the rightmost in the index after the GROUP BY columns after the normal where clauses. For e.g “where firstname=johnny GROUP BY city,country ORDER BY country” the index order should be (firstname,city,country).● Clustered Indexes : InnoDB’s clustered indexes actually store a B-Tree index and the rows together in the same structure.When a table has a clustered index, its rows are actually stored in the index’s leaf pages. The term “clustered” refers to the fact that rows with adjacent key values are stored close to each other.
Indexing Strategies for High Performance● Clustered Indexes : (contd.) InnoDB clusters the data by the primary key. If you don’t define a primary key, InnoDB will try to use a unique non-nullable index instead. If there’s no such index, InnoDB will define a hidden primary key for you and then cluster on that. InnoDB clusters records together only within a page. Pages with adjacent key values might be distant from each other.●
Indexing Strategies for High Performance● Clustered Indexes : (contd.) Example : SELECT * from user_info ORDER BY username. If our primary key is username then this querys output is very fast because it returns all the columns from the leaf node of the B-tree index only without referring the table and also since it is clustered on username hence rows are stored in a page in alphabetical order of the usernames hence ORDER BY does not require to do any sort in a single page.● If clustering on primary key is not desired i.e. If we do not need order by on primary key and then return almost all the columns, it is better not to define a primary key derived from any of the column values. For e.g. If we do not require queries as above then instead of defining primary key on username define primary key to be some user_id auto_increment because with username primary key there will be lots of random I/O in case of insertions (since insertions are not in any order of username) which is inefficient but with auto increment insertions follow sequential order thus saving random I/O.● MyISAM engine does not use clustering.
Indexing Strategies for High Performance● Covering Indexes : An index that contains all the data needed to satisfy a query is called a covering index. Consider the query : SELECT firstname, lastname from user_info where firstname=johnny and lastname like de%; The query is index covered since all the rows that are returned are part of the index (firstname, lastname, dob ).● Index covered queries are very fast since no row lookups (random I/O on disk) required, instead all rows returned from index.● Hash, spatial, and full-text indexes don’t use covering indexes, so MySQL can use only B-Tree indexes to cover queries.● When you issue a query that is covered by an index (an index-covered query), you’ll see “Using index” in the Extra column in EXPLAIN.● Due to the secondary index structure of InnoDB where secondary indexes store primary keys in their leaf nodes, queries that fetch columns that includes the primary key column and the secondary indexed columns is also a index covered query. For e.g SELECT user_id, firstname, lastname from user_info where firstname=johnny and lastname like de%; here user_id is not part of the index (firstname, lastname, dob ) but its a primary key so its index covered also.
Full-Text Searching● Most of the queries you’ll write will probably have WHERE clauses that compare values for equality, filter out ranges of rows, and so on. However, you might also need to perform keyword searches, which are based on relevance instead of comparing values to each other. Full-text search systems are designed for this purpose.● Full-Text search is based on finding words (terms) in documents instead of patterns.● For example we want to find all matching rows in the reviews table for which the reviews contains some or all words of the phrase “good excellent exciting”. ALTER TABLE reviews add FULLTEXT KEY(review); SELECT review, MATCH(review) AGAINST(good excellent exciting) as relevancy from reviews where MATCH(review) AGAINST(good excellent exciting);● Full-Text Searching can be accomplished without indexing also.● There are two different modes for Full-Text Searching : Natural Language Mode and Boolean Mode.● Only MyISAM engine supports Full-Text searching and indexing.
Full-Text Searching● Natural Language Mode of Full-Text Searching : The relevancy of a query with a particular row in the table is calculated as follows - 1. Compute the weight of each word/term in the fulltext indexed columns in each row. The weight for each word in a row increases if the number of times it occurs in one row increases and decreases if the number of rows it occurs in increases. i.e. to say that if a word in the query exists in few rows then that word determines how relevant that word is for ordering the search results. For example in the query “we had an exciting adventure” words such as “we”, “had” and “an” are pretty common terms in holiday reviews so they exists in more than 75% of rows in our database but words such as “exciting” and “adventure” are less common and occur in less than 10% of our database so “naturally we are looking for” rows in the table which contains words like “exciting” and “adventure” and thus they should be ranked higher. Infact words such as “we” , “an” , “the” etc. are called stopwords and they are not even considered while calculating weights. 2. Mathematically the formula for weight of a term ti in given row is given as : w[ti]= (log(dtf[ti])+1)/sumdtf * U/(1+0.0115*U) * log((N-nf[ti])/nf[ti])
Full-Text Searching● Natural Language Mode of Full-Text Searching : contd. 2. w[ti]= (log(dtf[ti])+1)/sumdtf * U/(1+0.0115*U) * log((N-nf[ti])/nf[ti]) where dtf[ti] : number of times term ti appears in the row. sumdtf : sum of (log(dtf)+1)s for all terms in the same row. U : number of unique terms in the row. N : Total number of rows. nf[ti] : number of rows that contain the term ti . The middle term signifies that if the length of the indexed columns in the row is shorter than the average length (= number of unique words) then the weight for that row increases (i.e. “short and sweet” row so as to say). 3. Then the Rank of that row is computed as R = ∑i w[ti]*qnf[ti], where qnf[ti] is the number of times the term ti occurs in the query. This value is given by SELECT MATCH() AGAINST() query.● Structure of the index : The index is a B-Tree structure with 2 levels. In the first level the nodes store the terms as keys and in the second level for each first level term, a pointer to the rows that contains the term. This is similar to inverted index.
Full-Text Searching● MATCH AGAINST clause cant be used to regard words from a particular column as more important than words from other columns. For example, you might want search results to appear first when the keywords appear in an reviews title.● Alternative solution to give twice the importance to the title of the review than the review itself : ALTER TABLE ADD FULLTEXT KEY(title, review); ALTER TABLE ADD FULLTEXT KEY(title); SELECT title, review, ROUND(MATCH(title, review) AGAINST(good excellent exciting), 3) AS full_rel, ROUND(MATCH(title) AGAINST(good excellent exciting), 3) AS title_rel FROM reviews WHERE MATCH(title, review) AGAINST(good excellent exciting) ORDER BY (2 * MATCH(title) AGAINST(good excellent exciting)) + MATCH(title, review) AGAINST(good excellent exciting) DESC;
Full-Text Searching● Boolean Mode of Full-Text Searching : In Boolean searches, the query itself specifies the relative relevance of each word in a match. When constructing a Boolean search query, you can use prefixes to modify the relative ranking of each keyword in the search string.● Examples : 1. SELECT * from reviews where MATCH(title,review) AGAINST (good ~bad +adventure in BOOLEAN MODE); i.e. Rows must contain the word adventure and rows with the word good should be ranked higher and rows with the word bad should be ranked lower. 2. SELECT * from reviews where MATCH(title,review) AGAINST (good -bad +adventure in BOOLEAN MODE); i.e. Rows must contain the word adventure and rows with the word good should be ranked higher and the rows should not contain the word bad. 3. SELECT * from reviews where MATCH(title,review) AGAINST (“good adventure” in BOOLEAN MODE); i.e. Rows should contain the phrase “good adventure”.
Full-Text Searching● Phrase searches tend to be quite slow. The full-text index alone can’t answer such queries, because it doesn’t record where words are located relative to each other in the original full-text collection. Consequently, the server actually has to look inside the rows to do a phrase search. To execute such a search, the server will find all documents that contain both “good” and “adventure” It will then fetch the rows from which the documents were built, and check for the exact phrase in the collection.● Disadvantages of Full-Text Indexing and Searching : 1. The index doesn’t record the indexed word’s position in the string, so proximity doesn’t contribute to relevance. 2. MySQL’s full-text indexing performs well when the index fits in memory, but if the index is not in memory it can be very slow, especially when the fields are large. 3. Modifying a piece of text with 100 words requires not 1 but up to 100 index operations.
Full-Text Searching● Disadvantages of Full-Text Indexing and Searching : (contd.) 4. The field length doesn’t usually affect other index types much, but with full- text indexing, text with 3 words and text with 10,000 words will have performance profiles that differ by orders of magnitude. 5. If there’s a full-text index and the query has a MATCH AGAINST clause that can use it, MySQL will use the full-text index to process the query. It will not compare the full-text index to the other indexes that might be used for the query. 6. The full-text search index can perform only full-text matches. Any other criteria in the query, such as WHERE clauses, must be applied after MySQL reads the row from the table. 7. Full-text indexes don’t store the actual text they index. Thus, you can never use a full-text index as a covering index. 8. Full-text indexes cannot be used for any type of sorting, other than sorting by relevance in natural-language mode. If you need to sort by something other than relevance, MySQL will use a filesort.
Full-Text Searching● Disadvantages of Full-Text Indexing and Searching : (contd.) SELECT * from reviews where MATCH(title, review) AGAINST (good exciting adventure) and review_id= 879; The query will not use the index on review_id since the preference for fulltext index is higher. So it will look into the fulltext index and filter out all matching rows then will use the review_id value to filter from WHERE clause. Solution: Index the review_id column also in the fulltext index by converting the values into some string format review_id_is_879. ALTER TABLE reviews add FULLTEXT KEY(review_id, title, review); SELECT * from reviews where MATCH(review_id, title, review) AGAINST (+review_id_is_879 good exciting adventure in BOOLEAN MODE);
References● Introduction to Algorithms, CLRS, 3rd Edition.● High Performance MySQL by Baron Schwartz, Peter Zaitsev and Vadim Tkachenko. Thank You