[PGCONF.EU 2017 Keynote] Database Horror Stories, Bad Code, and How Not to be a Statistic
•
1 like•1,343 views
36,321 tables, one row each, one database. A stored procedure with 1,752 parameters. A DBA convinced that storing everything in the entire database typed as BYTEA will definitely improve performance. All of these are true stories, and the sad fact is that many PostgreSQL professionals can relate to them -- even if they won't admit it in public. In this talk, I will discuss some of my favorite database anti-examples from The Daily WTF (including a particularly egregious one concocted by yours truly), and discuss how they came to be and where they came from. Most importantly, I'll cover how to avoid unintentionally creating your own database WTFs.
Recommended
More Related Content
Similar to [PGCONF.EU 2017 Keynote] Database Horror Stories, Bad Code, and How Not to be a Statistic
Similar to [PGCONF.EU 2017 Keynote] Database Horror Stories, Bad Code, and How Not to be a Statistic (20)
Recently uploaded
Recently uploaded (20)
[PGCONF.EU 2017 Keynote] Database Horror Stories, Bad Code, and How Not to be a Statistic
- 1. DATABASE HORRORS Alex Papadimoulis PostgreSQL Conference Europe 2017
- 2. WHO IS THIS GUY? ▪ From Tokyo, Japan ▪ よろしくお願いします ▪ Long-time relational advocate ▪ CELKO is a personal hero ▪ Inedo: Tools for DevOps ▪ Databases need DevOps, too! ▪ Bad-code Expert ▪ The Daily WTF
- 3. FORMAL DATABASE EDUCATION? Let’s do a quick survey!
- 5. LEARNED AS YOU DID?
- 6. (TYPICAL) RESULTS How did you learn? Formal Education Apprenticeship As I did
- 13. PROGRAMMER’S SAWHORSE ▪ Needlessly Complex ▪ Difficult to Build ▪ Structurally Deficient
- 16. RELIABILITY & REPEATABILITY ▪ Devices ▪ Tests ▪ Results ▪ Operators ▪ Readings
- 18. RUNNING THE NUMBERS… ▪ 1,000’s of Devices ▪ 10,000’s of Tests ▪ 100,000’s of Results ▪ 1,000,000’s of Operators ▪ 10,000,000’s of Readings
- 19. TEN MILLION READINGS ▪ In a Single Table!?!?!
- 25. 36,000 TABLES
- 27. RULES OF OPTIMIZATION 1. Don’t Do It 2. (For Experts Only) Don’t Do It Yet -- Michael A Jackson, 1974
- 33. RULES OF DYNAMIC SQL 1. Don’t Do It 2. (For Experts Only) Validate Your Input
- 36. RULES FOR HACKING PRODUCTION 1. Don’t Do It 2. (For Experts Only) Tell Everyone You Did It
- 38. RULES FOR CALENDAR TABLES 1. Don’t Use It 2. (For Experts Only) Only let experts use it
- 42. RULES FOR EXPERTS 1. You probably aren’t an Expert
- 45. Too Many Columns • Very High NULL-to-Data Ratio • Query complexity
- 46. Going Vertical
- 48. RULES FOR EXPERTS 1. You probably aren’t an Expert 2. You probably can’t judge Expertise
- 51. “There are too many layers”
- 53. “We’ll Have Too Many Stored Procs”
- 59. QUESTION YOUR QUESTIONS ▪ Why is it so hard to return HTML in SQL Queries ▪ Why is it so hard to make a more generic stored procedure? ▪ Why are transactions so needlessly complicated? ▪ Why isn’t there a built-in base-36 to base-10 converter?
- 63. THERE’S PROBABLY A SIMPLER WAY Thank you.
Editor's Notes
- Fairly unique to software
- Construction; a lot of people use it as an analogy Somewhat ironic since a lot software folks don’t work in the real world so much I’ve recently done a little Sawhorses are necessary
- One developer realized this Too expensive to buy; But easy to make Sketched it out and realized trig problem
- Not terribly complicated of a trig problem But one that needed a bit of work to work out
- found 16.26 degrees Disappointed that existing tools weren’t adequate Wood wasn’t accurate Cutting didn’t actually cut, it destroyed Angle calculations very difficult
- 36,000 tables, Querying basically impossible, Sooooo slow Little misunderstanding goes a loooong way General idea of how indexing worked More rows = more index lookups = more time You still have to lookup tables; those aren’t indexed quite as well
- Took all of the performance problems here Pushing them over there The problems were neatly stacked up and solved Tens of thousands of tables needing to be joined… not so much As it turns out, this is much slower And really, really, really stupid.
- Some might say this was premature optimization Root of all evil
- I am not alone There are a lot of well-intentioned developers who do horrible things
- DBCC TRACEON (1204)
- Calendar table
- I keep putting that in all of my slides But, who is an expert?
- George Carlin Most People think they’re smarter than the average Average Person thinks the Average Person is Dumb
- Story about The Database Expert I knew 25 years
- A lot of “expert” database people come across the looong table problem Any domain, but look at products Not everything has color
- Conclusion A lot of columns. A LOT of columns Confusing, hard to query Slower, since rows are so big
- What if we took all these columns and made them rows Near infinite number of “columns” and we forget about the null problem Could we take this furtheR? What if we have lots of tables? What’s easier to understand, one or many?
- you’ve effectively recreated a database inside of a database Not only did you reinvent the wheel, but you made it square And used a whole bunch of wheels to do it
- Evereyone, even experts can forget that technology is an evolution Relational databases evolved, experts don’t understand them Doesn’t everyone understand why a claw is on a hammer
- What appears to be a problem is actually a solution
- SQL Server is not a webserver It does not output HTML
- Stored Uber Procedure Large complex databases need lots of tables And lots of stored procs
- Transactions are complicated Isolation level, read commited, serializable Can barely keep them straight Not flexible enough
- All of these make sense if you don’t think about Not to say that there was no thought put in, just the wrong thought wrong places They’re really stupid in a really smart a way Tables-per-entity rehash
- Having lots of rows in table requires indexes, so… Stop right there Is it really slow? How slow? Are you sure? Ok… so, we’ll have a bunch of tables instead Is that faster? Really? Why? How?
- Don’t assume that you’re asking the right questions. Assume you’re asking the wrong questions. Story about why no built in Base-36 converter to store Canadian postal codes in zip column
- Poor assumptions lead to worse questions which lead to even worse answers, and thus the cycle repeasts. It reminds me of this wonderful quote. Heard of it?
- Opening line of the film, Peter Pan. It probably has been said before, time and time again. The same is true with the problems you’re solving; we are not pioneer and frontiersmen; we are solving well-known problems with well-known solutions We need to focus on learning what are problems, and what are solutions Otherwise we will just turn solutions into problems, and eventually end up with the inner platform effect
- Look around, are you actually in a rabit hole? It may not seem obvious, but question what you’re doing. OK, you’re building a CNC wood machine…. But Why not just buy one? Traditional equipment can’t do 16.26. Why? Calculated my angles need to be exactly this to fit. Why? Obviously I need to build saw horses.