Running a Realtime Stats Service on MySQL

Running a Realtime Stats Service on MySQL Cybozu Labs, Inc. Kazuho Oku

Background

Apr. 23 2009 Running Realtime Stats Service on MySQL

Who am I?

Name: Kazuho Oku ( 奥一穂 )

Original Developer of Palmscape / Xiino

The oldest web browser for Palm OS

Working at Cybozu Labs since 2005

Research subsidiary of Cybozu, Inc.

Cybozu is a leading groupware vendor in Japan

My weblog: tinyurl.com/kazuho

Introduction of Pathtraq

What is Pathtraq?

Started in Aug. 2007

Web ranking service

One of Japan’s largest

〜 10,000 users submit access information

〜 1,000,000 access infomation per day

like Alexa, but semi-realtime, and per-page

What is Pathtraq? (cont'd)

Automated Social News Service

find what's hot

like Google News + Digg

calculate relevance from access stats

Search by...

no filtering (all the Internet)

by category

by keyword

by URL (per-domain, etc.)

How to Provide Real-time Analysis?

Data Set (as of Apr. 23 2009)

# of URLs: 147,748,546

# of total accesses: 413,272,527

Sharding is not a good option

since we need to join the tables and aggregate

prefix-search by URL, search by keyword, then join with access data table

core tables should be stored on RAM

not on HDD, due to lots of random access

Our Decision was to...

Keep URL and access stats on RAM

compression for size and speed

Create a new message queue

Limit Pre-computation Load

Create our own cache, with locks

to minimize database access

Fulltext-search database on SSD

Our Servers

Main Server

Opteron 2218 x2, 64GB Mem

MySQL, Apache

Fulltext Search Server

Opteron 240EE, 2GB Mem, Intel SSD

MySQL (w. Tritonn/Senna)

Helper Servers

for Content Analysis

for Screenshot Generation

The Long Tail of the Internet

y=C ・ x -0.44

# of URLs with 1/10 hits: x2.75

Compressing URLs

Compressing URLs

The Challenges:

URLs are too short for gzip, etc.

URLs should be prefix-searchable in compressed form

How to run like 'http://www.mysql.com/%' on a compressed URL?

The Answer:

Static PPM + Range Coder

Static PPM

PPM: Prediction by Partial Matching

What is the next character after ".co"?

The answer is "m"!

PPM is used by 7-zip, etc.

Static PPM is PPM with static probabilistic model

Many URLs (or English words) have common patterns

Suitable for short texts (like URLs)

Range Coder

A fast variant of arithmetic compression

similar to huffmann encoding, but better

If probability of next character being "m" was 75%, it will be encoded into 0.42 bit

Compressed strings preserve the sort order of uncompressed form

Create Compression Functions

Build prediction table from stored URLs

Implement range coder

took an open-source impl. and optimized it

original impl. added some bits unnecessary at the tail

use SSE instructions for faster operation

coderepos.org/share/browser/lang/cplusplus/range_coder

Link the coder and the table to create MySQL UDFs

Rewriting the Server Logic

Change schema

url varchar(255) not null # with unique index

↓

urlc varbinary(767) not null # with unique index

Change prefix-search form

url like 'http://example.com/%'

↓

url_compress('http://example.com/')<=urlc and urlc<url_compress('http://example.com 0 ')

Note: "0" is next character of '/'

Compression Ratio

Compression ratio: 37%

Size of prediction table: 4MB

Benchmark of the compression functions

compression: 40MB/sec. (570k URLs/sec.)

decompression: 19.3MB/sec. (280k URLs/sec.)

fast enough since searchable in compressed form

Prefix-search became faster

shorter indexes lead to faster operation

Re InnoDB Compression

URL Compression can coexist with InnoDB compression

though we aren't using InnoDB compression on our production environment

Apr. 23 2009 Running Realtime Stats Service on MySQL Compression Table Size N/A 100% URL compression 57% InnoDB compression 50% using both 33%

Compressing the Stats Table

Used to have two int columns: at , cnt

it was waste of space, since...

most cnt values are very small numbers

most accesses to each URL occur on a short period (ex. the day the blog entry was written)

at field should be part of the indexes

Apr. 23 2009 Running Realtime Stats Service on MySQL at (hours since epoch) cnt (# of hits) 330168 1 330169 2 330173 1 330197 1

Compressing the Stats Table (cont'd)

Merge the rows into a sparse array

example on the prev. page becomes:

(offset=330197),1,0(repeated 23 times),1,2,1

Then compress the array

the example becomes a blob of 8 bytes

originally was 8 bytes x 4 rows with index

And store the array in a single column

fewer rows lead to smaller table, faster access

Compressing the Stats Table (cont'd)

Write MySQL UDFs to access the sparse array

cnt_add(column,at,cnt)

-- adds cnt on given index (at)

cnt_between(column,from,to)

-- returns # of hits between given hours

and more...

We use int[N] arrays for vectorized calc.

especially when creating access charts

Create a new Message Queue

Q4M

A simple, reliable, fast message queue

runs as a pluggable storage engine of MySQL

GPL License; q4m.31tools.com

presented yesterday at MySQL Conference :-p

slides at tinyurl.com/q4m2009

Used for relaying messages between our servers

Limiting Pre-computation Load

Limit # of CPU-intensive Pre-computations

Use cron & setlock

setlock is part of daemontools by djb

setlock

serializes processes by using flock

-n option: use trylock; if locked, do nothing

# use only one CPU core for pre-computation

*/2 * * * * setlock –n /tmp/tasks.lock precompute_hot_entries

5 0 * * * setlock /tmp/tasks.lock precompute_yesterday_data

Limit # of Disk-intensive Pre-computations

Divide pre-computation to blocks and sleep depending on the elapsed time

my $LOAD = 0.25;

while (true) {

my $start = time();

precompute_block();

sleep(min(time - $start, 0) * (1 - $LOAD) / $LOAD);

}

Creating our own Cache System

The Problem

Query cache is flushed on table update

access stats can be (should be) cached for a certain period

Memcached has a thundering-herd problem

all clients try to read the database when a cached-entry expires

critical for us since our queries does joins, aggregations, and sort operations

Swifty and KeyedMutex

Swifty is a mmap-based cache

cached data shared between processes

lock-free on read, flock on write

notifies a single client that the accessed entry is going to expire within few seconds

notified client can start updating a cache entry before it expires

KeyedMutex

a daemon used to block multiple clients issuing same SQL queries

Swifty and KeyedMutexd (cont'd)

Source codes are available:

coderepos.org/share/browser/lang/c/swifty

coderepos.org/share/browser/lang/perl/Cache-Swifty

coderepos.org/share/browser/lang/perl/KeyedMutex

Fulltext-search on SSD

Senna / Tritonn

Senna is a FTS engine popular in Japan

might not work well with European languages

Tritonn is a replacement of MyISAM FTS

uses Senna as backend

faster than MyISAM FTS

Wrote patches to support SSD

during our transition from RAM to SSD

patches accepted in Senna 1.1.4 / Tritonn 1.0.12

FTS: RAM-based vs. SSD-based

Size of FTS data: 〜 20GB

Downgraded hardware to see if SSD-based FTS is feasible

Speed became ¼

but latency of searches are well below one second

Apr. 23 2009 Running Realtime Stats Service on MySQL Old Hardware New Hardware CPU Opteron 2218 (2.6GHz) x2 Opteron 240 (1.4GHz) Memory 32GB 2GB Storage 7200rpm SATA HDD SSD (Intel X25-M)

Summary

Summary

Use UDFs for optimization

Sometime it is easier to scale UP

esp. when you can estimate your data growth

Use SSD for FTS

Baidu (China's leading search engine) uses SSD

Most of the things introduced are OSS

We plan to open-source our URL compression table as well

We are Looking for...

If you are interested in localizing Pathtraq to your country, please contact us

we do not have resources outside of Japan

to translate the web interface

to ask people to install our browser extension

to follow local regulations, etc.

Thank you for listening

tinyurl.com/kazuho

Running a Realtime Stats Service on MySQL

0 comments

Notes on slide 1

5 Favorites