Big Bad "Upgraded" Postgres

Big Bad “Upgraded” Postgres

Robert Treat

/ Presentation

Wednesday, May 23, 12

Intro

• Robert Treat
• xzilla.net
• @robtreat2
• +RobertTreat


Intro, part 2

• COO @ OmniTI
• Full Stack Tech Consulting
• Remote Database Management and Consulting
• Large Scale / Mission Critical
• We’re Hiring!


Philosophy

OmniTI has a reputation for scalable web
applications and architectures.

We didn't learn this stuff overnight.

Like many success stories, we acquired
experience through trial and error, constant
collaboration between development and
operations teams, and an unwavering commitment
to excellence.

But we still lean on our friends and peers to see
how things can be done better.

Philosophy


So, What Is Big Bad Postgres?

• Original Project
• Convert TB+ Sized Oracle ODS To Postgres
• 2005, Postgres 8.1


OLTP Instance:
Drives the Site
Warm Standby
Oracle
Oracle

mysql Data Warehouse
logging, bulk processing
Oracle
mysql


OLTP Instance:
Drives the Site
Warm Standby
Oracle
Oracle

mysql Data Warehouse
logging, bulk processing

XXX
Postgres
mysql



ProTip

• Remove Oracle
Licensing Costs

• Remove Oracle Add-On
Costs



ProTip BroTip

• Remove Oracle
• Missing features
Licensing Costs

• Remove Oracle Add-On • Upgrades are a
problem
Costs



• Missing Features (Postgres 8.1)
• Heterogeneous Replication (dbi-link)
• Autonomous Transactions (dblink)
• Backups (zfs)
• Aggregate SQL (plpgsql)
• Large Selects (cursors?)
• Upgrades (?)



• Save $500K in Licenses
• $100K in labor
• Took Roughly 6 Months
• Built lots of useful tools
• Learned a lot about Postgres
• Everybody’s happy

http://lethargy.org/~jesus/writes/big-bad-postgresql


Big Bad “Broken” Postgres

• Feb 2008, disaster struck
• disk failures
• memory issues
• software bugs
• no failover box!
• disaster recovery ensued



Using ZFS snapshots, we were able to modify Postgres
code, deploy / test on production copy of data, and
eventually get a running system.



• Fallout:
• New Machines (2 of them!)
• Upgrade to Postgres 8.3
• given corrupt data, dump/restore was forced
• made life easier operationally
• Happiness restored!


OLTP Instance:
Drives the Site
Warm Standby
Oracle
Oracle

“identical”
Data Warehouse
Postgres
Postgres 8.3
8.3



robert@omniti.com at 12:24 on 2009-02-17
8.4 is approaching beta, and has some good
features we could take advantage of on pgods,
if we could get it to upgrade.



• Issues to handle
• 3TB, compressed
• Limited by hardware

• Limited by spare time



• Options?



• Options?
• SLONY - Well known, but mostly unusable
solution (issues with partitioned tables are
primary culprit, but lots of other dynamic ddl as
well)



• Options?
well)

• Mammoth Replicator - can replicate between
versions, but slave creation is a problem.



• Options?
well)

• Mammoth Replicator - can replicate between
versions, but slave creation is a problem.

• “8.4 dev tree has a pg_migrator script in it, for
in place upgrades, this should be investigated
in our environment.”



• 8.4 dev pg_migrator
• Ran into bugs with Solaris support during dev
• "file.c", line 592: warning: argument #4 is incompatible with
prototype:

• Spent ~ 1 month on this before it got put on the
back burner



• 8.4.0 pg_migrator (Aug 2009)
• packages all built without major issues

• 8.4.0 had a bug with indexes & plperl
• so, we delayed

• actually, we postponed



• 9.0 upgrade?
• Jan 2011, 9.0 is in development

• Given months of testing, we focused on 9.0



• Some tools we use:
• dblink
• plperl / dbilink
• fuzzystring
• freespacemap
• pg_reorg
• secure check postgres



• SNAG #1
• can’t symlink across filesystems
• (aka zfs datasets)



• SNAG #1
• can’t symlink across filesystems
• (aka zfs datasets)

Given $PGDATA = /pgsql/main

NO: YES:
/pgsql/main /pgsql/main/83
/pgsql/main9 /pgsql/main/90


• SNAG #2
• un-upgradable data types


• SNAG #2
• left over pg_reorg cruft
• (special table data types)


• SNAG #2, part b


• SNAG #2, part b
• “NAME” type
• check_postgres.pg_stat_activity.datname


• SNAG #2, part c
• “NAME” type


• SNAG #2, part c
• “NAME” type

create table x as
select
tablename,
pg_relation_size(schemaname||’.’||tablename)
from
pg_tables


-- plan of attack --
swap in minimal cron / conﬁgs
shut down 8.3 database
zfs snapshot fs
bring up 8.3 / 9.0 databases
run pg_upgrade
verify migration complete
turn on snap replication
sanity checking
slow role more services back on-line
back in buisness


-- actual attack --
swap in minimal cron / conﬁgs
shut down 8.3 database
zfs snapshot fs
bring up 8.3 / 9.0 databases
run pg_upgrade
**hit a bug in role creation**
discuss
rollback
(rename the control ﬁle of the old cluster)
bring back up 8.3 database



CREATE ROLE asha;
ALTER ROLE asha SET role TO
'omniti';
.. sometime later ...
CREATE ROLE omniti;



Added to TODO:

! Allow pg_dumpall to output
restorable ALTER USER/DATABASE
SET settings

March 2011



We could have done something where we dropped or
modiﬁed all roles and recreated them after upgrade, but
this didn’t seem like the right ﬁx.



We could have done something where we dropped or
modiﬁed all roles and recreated them after upgrade, but
this didn’t seem like the right ﬁx.

eventually, we started working on a patch



Meanwhile, at another client, not far away...



• more pg_upgrade bugs
• Fix pg_upgrade's handling of TOAST tables
(april)

• Fix pg_upgrade to preserve toast tables'
relfrozenxids during an upgrade from 8.3
(sept)



• more pg_upgrade bugs
• Fix pg_upgrade's handling of TOAST tables
(april)

• Fix pg_upgrade to preserve toast tables'
relfrozenxids during an upgrade from 8.3
(sept)

Not directly relevant to “Big Bad”,
but did take time / energy away,
and was not conﬁdence inspiring



• back to the role bug
• after ~ 3 months, the patch was accepted
(oct)

• 9.1 is now just around the corner, so...



• back to the role bug
• after ~ 3 months, the patch was accepted
(oct)

• 9.1 is now just around the corner, so...

we decided to go to 9.1



• 9.1 released in September, 2011
• 9.1 upgrade testing begins

• schema tests
• compatibility tests



• 9.1 released in September, 2011
• 9.1 upgrade testing begins

• schema tests
• compatibility tests

2011-10-25
Judgement Day



• doh #1
• non-empty tablespace
• left over from some 9.1 testing (pg_dump /
pg_restore of schema only)



• doh #1
• non-empty tablespace
• left over from some 9.1 testing (pg_dump /
pg_restore of schema only)

Creating databases in the new cluster
psql:/pgdata/main/pg_upgrade_dump_globals.sql:247: NOTICE: schema "ods" does not exist
psql:/pgdata/main/pg_upgrade_dump_globals.sql:255: NOTICE: schema "check_postgres" does not exist

psql:/pgdata/main/pg_upgrade_dump_globals.sql:303: ERROR: directory "/
pgdata/alldata1/PG_9.1_201105231" already in use as a tablespace

There were problems executing "/opt/pgsql911/bin/psql" --set
ON_ERROR_STOP=on --no-psqlrc --port 5491 --username "postgres" -f "/
pgdata/main/pg_upgrade_dump_globals.sql" --dbname template1 >> "/dev/null"
Failure, exiting



• doh #2
• new bad data types had snuck in (more
name, some unknown)

• drop tables, alter data types



• doh #3
• actually it worked*



• doh #3

• begin brining services back online
• analyze, vacuum

• turn on “snap job” replication

• turn on regular replication



• doh #3

• begin brining services back online
• analyze, vacuum

• turn on “snap job” replication

• turn on regular replication
• aka “the point of no return”

* don’t worry, it didn’t really work, it just looked like it did


• Replication was running
• 100’s of tables
• ETL services not yet on
• dinner time :-)



Things Were Sailing Along Fine



“if this were the movie titanic, this is
the scene where you see the two
guys up in the watch tower joking
around before the iceberg hits.”



NOTICE: ERROR: value too long for type character
varying(40) at line 96.



NOTICE: ERROR: value too long for type character
varying(40) at line 96.

normally we see this on (oracle) number -> (pgsql) integer



• Verified function with debug output
• function creates a temp table
matching primary table
• copies replicated data into temp
table
• replaces data in actual table with
data from temp table
• This was all working



• function with same data working on
the other server
• verified sql ran fine using dblink
(so, plperl specific)



I hate encoding issues



• Ruled out several config options
(lc_*, locale on the machine)
• plperl running on different version
of libpq?



of libpq?
• install libpq5 built on 9.1

• install dbdpg built on new libpq



of libpq?
• install libpq5 built on 9.1

• install dbdpg built on new libpq

...no



• wrote perl script that used dbd:pg
to grab data from oracle
• worked!

• so, plperl eh?


postgres=# show client_encoding;
client_encoding
-----------------
UTF8
(1 row)


client_encoding
-----------------
UTF8
(1 row)

postgres=# select * from dbi_link.remote_select(3,'select chr(255) as
bar') t (bar text);
bar
-----
Ã¿
(1 row)


client_encoding
-----------------
UTF8
(1 row)

bar') t (bar text);
bar
-----
Ã¿
(1 row)

postgres=# set client_encoding = latin1;
SET
bar') t (bar text);
bar
-----
ÿ
(1 row)



• check lang var

• pg user

• root

• smf init script

• everything checked out :-


“But Wait, Maybe It Is The LC Stuff”
-bash-3.00$ sudo pargs -e 17302
Password:
17302: /opt/pgsql8314/bin/postgres -D /pgdata/main
envp[0]: LC_TIME=C
envp[1]: LC_NUMERIC=C
envp[2]: LC_MONETARY=C
envp[3]: LC_MESSAGES=en_US.UTF-8
envp[4]: LC_CTYPE=en_US.UTF-8
envp[5]: LC_COLLATE=en_US.UTF-8
envp[6]: LD_LIBRARY_PATH=/opt/oracle/amd64
envp[7]: ORACLE_HOME=/opt/oracle
envp[8]: PATH=/usr/sbin:/usr/bin
envp[9]: PERL5LIB=/data/CPAN/lib/site_perl
envp[10]: PGDATA=/pgdata/main
envp[11]: PGPREFIX=/opt/pgsql
envp[12]: PGSYSCONFDIR=/opt/pgsql8314/etc
envp[13]: PGUSER=postgres
envp[14]: SMF_FMRI=svc:/database/postgres:default
envp[15]: SMF_METHOD=/opt/pgsql/bin/pg_ctl -D $PGDATA
start -w
envp[16]: SMF_RESTARTER=svc:/system/svc/restarter:default
envp[17]: TNS_ADMIN=/opt/oracle/network/admin
envp[18]: TZ=US/Eastern



-bash-3.00$ sudo pargs -e `pgrep postgres` |
grep LC_CTYPE
Password:



-bash-3.00$ sudo pargs -e `pgrep postgres` |
grep LC_CTYPE
Password:

note: this is actually from the “broken” one



After more experimentation, including restart of
database with adjusted LANG and en vars, I was
able to fix the odd LC settings, but not the
remote data select with plperl...

#DOH



have you ﬁgured it out yet?



pg_enable_utf8



$dbh->{pg_enable_utf8} = 1;

Force strings passed to and from plperl to be in UTF8 encoding.
String are converted to UTF8 on the way into perl
and to the database encoding on the way back. This
avoids a number of observed anomalies, and ensures
Perl a consistent view of the world.

https://github.com/postgres/postgres/commit/50d89d422f9c68a52a6964e5468e8eb4f90b1d95



$dbh->{pg_enable_utf8} = 1;

Actually a change in 9.0
http://www.postgresql.org/docs/9.1/static/release-9-0.html
* Verify that PL/Perl return values are valid in the server
encoding (Andrew Dunstan)

Note: Perl may otherwise make
assumptions that your data is Latin1



My 2:00AM Summary...

“In any case, this seems like a horrible backwards
compatibility nightmare thats likely to eat peoples data; I
only noticed it because I was pulling data from a varchar(20)
into a varchar(20) and it complained the data size was too
long. Had I been using text (which is what I normally do), I
think I would have screwed myself.”



One last broken job...

SELECT
42, cntry_abbr as country,
date_trunc('day', h.hitdate) as rollup_day,
pg.price_group_id,
count(1) as hits
FROM
tblhits h
join tbladvertiser_campaign sc on h.partner = sc.source_code
join tblcountry flc on promo.perl_geo_ip_country(h.ipaddress) = flc.cntry_abbr
left join tbladvertiser_price_groups pg on pg.campaign_id = 42
and h.hitdate between pg.start_date and coalesce(pg.end_date, h.hitdate)
and get_bit(decode(pg.countries::text, 'hex'), flc.country_id::integer) > 0
WHERE
h.hitdate >= '2011-10-26'::date and sc.campaign_id = 42
and h.hitdate < '2011-10-31'::date + '1 day'::interval
GROUP BY
flc.country_abbreviation, date_trunc('day', h.hitdate), pg.price_group_id;




Default output of bytea columns was changed
in Postgres 9.1




SELECT
42, cntry_abbr as country,
date_trunc('day', h.hitdate) as rollup_day,
pg.price_group_id,
count(1) as hits
FROM
tblhits h
join tbladvertiser_campaign sc on h.partner = sc.source_code
join tblcountry flc on promo.perl_geo_ip_country(h.ipaddress) = flc.cntry_abbr
left join tbladvertiser_price_groups pg on pg.campaign_id = 42
and h.hitdate between pg.start_date and coalesce(pg.end_date, h.hitdate)
and get_bit(pg.countries, flc.country_id::integer) > 0
WHERE
h.hitdate >= '2011-10-26'::date and sc.campaign_id = 42
and h.hitdate < '2011-10-31'::date + '1 day'::interval
GROUP BY
flc.country_abbreviation, date_trunc('day', h.hitdate), pg.price_group_id;


Aftermath

• did utf8 issue cause data corruption?

• extensive post-upgrade testing

• “data diff” between servers

• fixed dozen of problems


Aftermath

• did utf8 issue cause data corruption?

• extensive post-upgrade testing

• “data diff” between servers

• fixed dozen of problems

NONE attributable to the upgrade or plperl issues


Aftermath

pg_upgrade works pretty well


Alls Well That Ends Well

We upgraded the other box



Took ~ 45 minutes



Nothing broke



Yet?

;-)


THE END

Thanks!
PGCon

omniti dba team
postgres hackers

Slides
http://www.xzilla.net/

@robtreat2


Big Bad "Upgraded" Postgres

More Related Content

Similar to Big Bad "Upgraded" Postgres

More from Robert Treat

Recently uploaded

Big Bad "Upgraded" Postgres