Addressing vendor weaknesses in user space (Robert Treat)

Addressing Vendor Weaknesses in User-Space

ROBERT TREAT,
OmniTI

Highload++ 2011

@robtreat2
xzilla.net
+Robert Treat
1
Monday, October 3, 11

Who Am I?

OMNTI - Internet Scalability Consultants
Lead Database Operations

2

Who Am I?


“Large Scale”

3

Who Am I?


“Large Scale”

High Transactions
TB+ Data

4

Who Am I?


“Large Scale”

High Transactions
TB+ Data

Mission Critical

5

Who Am I?

Database Operations @OMNTI

Postgres
MySQL
Oracle
& More

6

Postgres for Scalability

Traditional RDBMS
Highly Extensible
Runs Everywhere
Talks To Everything
“BSD” Licensed
15+ Years Development
Open Development Community

7

The Bloat Problem

Data Footprint Can Be Critical To Performance

8

The Bloat Problem


Size On Disk Affects The Needs Of
RAM, Disk Speed, Storage

9

The Bloat Problem



“Bloat” is unused, wasted disk space,
used by the database,
but not needed for actual data storage

10

The Bloat Problem



“Bloat” is unused, wasted disk space,
taken up by the database,
but not needed for actual data storage

Why?
11

MVCC Architecture

Multiversion Concurrency Control (MVCC) allows
Postgres to offer high concurrency even during
signiﬁcant database read/write activity. MVCC
speciﬁcally offers behavior where "readers never block
writers, and writers never block readers".

12

MVCC Architecture

• Oracle
• MySQL (InnoDB)
• Informix
• Firebird
• MSSQL (optional)

13

MVCC Architecture

• Oracle
• MySQL (InnoDB)
• Informix
• Firebird
• MSSQL (optional)
• CouchDB

14

“Bloat” Manifests Differently, But Is Common

• MongoDB (deletes, some updates)
• dump/restore
• mongod --repair
• db.runCommand( { compact : 'mycollectionname' } )

• Lucene (updates)
• Hadoop / HDFS (small ﬁles)

15

Postgres MVCC Architecture

• Implemented Postgres 6.5
• 1999, Vadim Mikheev
• MVCC Unmasked
• http://momjian.us/main/writings/pgsql/mvcc.pdf

16


• Postgres maintains global transaction counters
• Keeps track of transaction counter per row for
• creating transaction
• removing transaction
• Using these counters, Postgres allows different
transactions to see different rows, based on visibility rules.

17


• Postgres maintains global transaction counters
• Keeps track of transaction counter per row for
• creating transaction
• removing transaction
• Using these counters, Postgres allows different
transactions to see different rows, based on visibility rules.

Transaction Reading An Old Row
Doesn’t Block Transaction Writing A Row
18

MVCC Architecture

user_id X42
Create 32 INSERT
Expire

19

MVCC Architecture

user_id X42
Create 32 INSERT
Expire

user_id X42
Create 32 DELETE
Expire 38

20

MVCC Architecture

user_id X69
Create 43 OLD(delete)
Expire 56
user_id X69
UPDATE
Create 43 NEW(insert)
Expire

21

MVCC Architecture

user_id X69
Create 43 <~~ DEAD ROW
Expire 56
user_id X69
Clean Up / Bloat
Create 43 <~~ VISIBLE ROW
Expire

22

MVCC Architecture

user_id X69
Create 43 <~~ DEAD ROW
Expire 56
user_id X69
Clean Up / Bloat
Create 43 <~~ VISIBLE ROW
Expire

Speed Up SQL Commands By
Dealing With Clean Up Later

23

How Postgres Deals With Bloat

• Heap-Only-Tuples (HOT)
• On-The-Fly, Per Page Cleanup
• Marks Given Row’s Space Reusable
• Update Only

24

How Postgres Deals With Bloat

• Heap-Only-Tuples (HOT)
• On-The-Fly, Per Page Cleanup
• Marks Given Row’s Space Reusable
• Update Only
• VACUUM
• Non-Blocking Bulk Cleanup
• Removes End-Of-File Pages
• “autovacuum” Process Monitors Tables

25

Problems With Automatic Cleanup

• HOT
• Update Only
• Doesn’t Work With Changing Index Data

26

Problems With Automatic Cleanup

• HOT
• Update Only
• Doesn’t Work When Changing Index Data
• VACUUM
• Must Wait For Long Transactions To Complete
• Costs I/O, Can Only Work So Fast
• Can’t Remove Non End-Of-File Pages
• Leaves A “High Water Mark”

27

Dealing With Bloat - The Hard Way

• VACUUM FULL / CLUSTER
• The Good
• Reclaims All “Dead Rows”

28

Dealing With Bloat - The Hard Way

• VACUUM FULL / CLUSTER
• The Good
• Reclaims All “Dead Rows”
• The Bad
• Exclusive Lock
• Rewrite All Data In Tables
• Needs Working Space
• Heavy I/O

29

Monitoring Your Bloat

• check_postgres.pl
• Nagios plugin
• Compares physical size to row size estimates
• http://bucardo.org/wiki/Check_postgres
• “bloat report”
• Script to measure table/index bloat
• Compares physical size to row size estimates
• http://labs.omniti.com/labs/pgtreats/
browser/trunk/tools/

30

Dealing With Bloat In Userspace

• Solving MVCC Bloat Is A “Hard Problem”
• Even a good solution would be hard to
implement in core

31

Dealing With Bloat In Userspace

• Solving MVCC Bloat Is A “Hard Problem”
• Even a good solution would be hard to
implement in core

• Can we build a tool in user space?
• Develop solution quicker
• Easier to deploy and maintain
• Provide a prototype for future development

32

Dealing With Bloat Redux

• Updating A Row Rewrites Data To New Location

33


• Use Vacuum To Mark Old Rows “Reusable”

34


• Update Row To Rewrite Data At “Front” Of Page

35


• Use Vacuum To Reclaim Space From End Of File

36


• Use Vacuum To Reclaim Space From End Of File
• Put A Script On It
• https://labs.omniti.com/pgtreats/trunk/tools/compact_table

37


• “Compact Table”
• Requires Lots of Time, I/O
• Often Causes Heavy Index Bloat
• Heavy Concurrency Bloats Faster Than
We Can Recover It

38

Dealing With Bloat For Real!

• Enter “pg_reorg”

39


• Vacuum / Cluster Replacement

40


• Command Line Tool

41


• Online Table Rewrite
• Uses Minimal Locking

42


• Developed By NTT

43


• Developed By NTT
• BSD Licensed
• C Code
• http://pgfoundry.org/projects/reorg/

44

How pg_reorg Works

• Create a log table for changes
• Create triggers on the old table to log changes (I/U/D)
• Create a new table with a copy of all data in old table
• Create all indexes on the new table
• Apply all changes from the log table to the new table
• Modify the system catalogs information about table ﬁles
• Drop old table, leaving new table in it’s place

45

How pg_reorg Works

• Create a log table for changes
• Create triggers on the old table to log changes
• Create a new table with a copy of all data in old table
• Create all indexes on the new table
• Apply all changes from the log table to the new table
• MODIFY THE SYSTEM CATALOGS
INFORMATION ABOUT THE TABLE FILES (!!!)
• Drop old table, leaving the new table in it’s place

46


Open Source Code

The Power Is In Your Hands

Look At Code
Examine the SQL
(User Space Is Really Visible)

TEST!

47


What Does Testing Look Like?

Create Some Tables,
Create Artiﬁcial Bloat,
run pg_reorg

48


What Does Testing Look Like?

Create Some Tables,
Create Artiﬁcial Bloat,
run pg_reorg

WIN!

49


Test In “Prod”

50


Test In “Prod”

Find Some Bloated Tables,
Make Backup Of Tables,
Cross Fingers,
pg_reorg

51


Test In “Prod”

Find Some Bloated Tables,
Make Backup Of Tables,
Cross Fingers,
pg_reorg

WIN!

52


Eventually You Have To Use It
On Something That Matters

53

pg_reorg In The Real World

• Production Database (OLTP)
• 540GB Size
• 2000 TPS (off-peak time, multiple statements)
• Largest Table (pre-reorg) 127GB

54


• Production Database (OLTP)
• 540GB Size
• 2000 TPS (off-peak time, multiple statements)
• Largest Table (pre-reorg) 127GB

• Rebuild Stats
• 5.75 Hours To Rebuild
• Reclaimed 52GB Disk Space
• No outages reported for Website/API’s

55


56


57


YAY!

58

Return Of
The Jedi

59

“your overconfidence is
your weakness.”

-Luke Skywalker

60

“your faith in your
friends is yours.”

-Emperor Palpatine

61

Sometimes You Can Have Both

Trust in NTT’s Code == faith in friends

Success in production == overconﬁdence

62

When Good pg_reorgs Go Bad!

WARNING: unexpected attrdef record found
for attr 61 of rel orders

WARNING: 1 attrdef record(s) missing for rel
orders

63


WARNING: unexpected attrdef record found
for attr 61 of rel orders

WARNING: 1 attrdef record(s) missing for rel
orders

Yes, On A Production System
Yes, Trying To Take 1000’s of Orders Per Second

64


create table test (
a int4,
b int4 default 2112,
c bool
);

65


create table test (
a int4,
c bool
);

Postgres internals track defaults / constraints
based on column position “2”, not column name “b”

66


create table test (
a int4,
c bool
);


If you drop column “a” and then do pg_reorg, column
“c” is now column “2”, and default 2112 is on boolean

67


create table test (
a int4,
c bool
);


If you drop column “a” and then do pg_reorg, column
“c” is now column “2”, and default 2112 is on boolean

This Is Fair - pg_reorg hacks the system tables
68


Basic Fix: Drop All Defaults And Recreate

69



Alternative Fix: Hack System Catalogs Some More

70



Alternative Fix: Hack System Catalogs Some More

Haven’t we had enough
system catalog hacking
for now?

71


“now, if you'll excuse me,
I'll go away and have a
heart attack.”

72

What Next?

Report Problem To Mailing List
Submit A Patch
Ultimately The Problem Is Fixed
Everyone’s Happy?

73

Hackers Discussion
Postgres Development Community Is Funny

Sometimes Hard To Get Them To
Recognize Problems

Not Everyone See Online Rebuild As A Big Problem

74

Hackers Discussion
Postgres Development Community Is Funny

Sometimes Hard To Get Them To
Recognize Problems

Not Everyone See Online Rebuild As A Big Problem

In All The Fairness,
Not Everyone Has This Problem

75

Hackers Discussion
Hackers Meeting 2011,
Discussion On Internal Queuing System

Could Be Used As Underlying Basis For
On-Line Rebuilding

Until Then...

76

pg_reorg Is A Great Tool!
Best Option For Difﬁcult Situation

Just Be Careful!

77

THANKS!

Highload++
NTT
OmniTI
Postgres Community
Momjian, Depesz, Patel, Kocoloski

xzilla.net
@robtreat2
+ Robert Treat
78

Addressing vendor weaknesses in user space (Robert Treat)

Recommended

Recommended

More Related Content

Similar to Addressing vendor weaknesses in user space (Robert Treat)

Similar to Addressing vendor weaknesses in user space (Robert Treat) (20)

More from Ontico

More from Ontico (20)

Recently uploaded

Recently uploaded (20)

Addressing vendor weaknesses in user space (Robert Treat)