On active tables with foreign key constraints, high volume in-database inserts to the table may take prohibitively long to complete, even when run in autocommit mode. As a solution, we investigated the relative performance of extracting, then reinserting the data using client-side COPY commands. We compare the performance of table-to-table inserts, inserts into unlogged tables, and our client-side copies. As presented at the 2017 New York 2nd Quadrant Postgres Conference.

1. Round Trip Client-Side COPY
for High Volume Postgres
Inserts
DANIEL GERLANC
PRESIDENT & FOUNDER
Enplus Advisors, Inc.

2. Round Trip Client-Side COPY (RTC COPY)
2
COPY to buffer on client, then COPY into destination
table

3. Problem
3
Insert large number of records from SELECTed tables into a
logged destination table
Don’t control the schema
If possible, all or nothing

4. Problem
4
Table(s)-to-Table inserts were not completing!

5. Solution:
5
1. INSERT in smaller batches
2. Round Trip Client-Side COPY (RTC Copy)

6. How’d you come up with this?
6
To avoid putting load on a production server I ran some
analyses on my local Postgres instance
COPYed from my local Postgres instance to disk and COPYed
to the remote Postgres
Noticed that I never had problems inserting into the
production database this way

7. Benchmark
7

8. Benchmark
8
Always 1..10K2
Insert 1..n11
Start With = count(n_users) * 53
Insert: count(n_users) * 54
T2T INSERT or RTC COPY5

9. Benchmark
9
1. Drop and recreate all tables except benchmarks result table
2. Populate users
3. Populate categories
4. Populate user_stats with random data for all users, 5 categories
5. Populate user_stats_staging with random data for all users and
additional 5 categories
6. Run vacuum analyze
7. Copy records from user_stats_staging to user_stats using
INSERT or COPY

10. Benchmark
10
e.g. Number of Users = 1K
Logged Unlogged
T2T INSERT
RTC COPY
30 Repetitions for each condition
n_users = (1K, 10K, 50K, 100K, 500K)

11. 1K Users, 5K Existing, 5K Insert
11
Box Plot of 30 Repetitions per Condition

12. 1K Users, 5K Existing, 5K Insert
12
BCa Bootstrap Confidence Intervals for
Mean Difference, 2,000 Replications*
*Calculated with R 3.4.2 and bootES 1.2

13. 10K Users, 50K Existing, 50K Insert
13
Box Plot of 30 Repetitions per Condition

14. 10K Users, 50K Existing, 50K Insert
14
BCa Bootstrap Confidence Intervals for
Mean Difference, 2,000 Replications*
*Calculated with R 3.4.2 and bootES 1.2

15. 50K Users, 250K Existing, 250K Insert
15
Box Plot of 30 Repetitions per Condition

16. 50K Users, 250K Existing, 250K Insert
16
BCa Bootstrap Confidence Intervals
for Mean Difference, 2,000 Replications*
*Calculated with R 3.4.2 and bootES 1.2

17. 100K Users, 500K Existing, 500K Insert
17
Box Plot of 30 Repetitions per Condition

18. 100K Users, 500K Existing, 500K Insert
18
BCa Bootstrap Confidence Intervals
for Mean Difference, 2,000 Replications*
*Calculated with R 3.4.2 and bootES 1.2

19. 500K Users, 2.5M Existing, 2.5M Insert
19
Box Plot of 30 Repetitions per Condition

20. 500K Users, 2.5M Existing, 2.5M Insert
20
BCa Bootstrap Confidence Intervals
for Mean Difference, 2,000 Replications*
*Calculated with R 3.4.2 and bootES 1.2

21. Is the difference meaningful?
21
Mean and BCa Confidence Intervals, 2,000 Replications

22. Statistically Significant Results
22
5K Records: No statistically significant difference
50K Records: Logged COPY faster
100K Records: Unlogged COPY faster
500K Records: Unlogged INSERT faster
Other differences not statistically significant.

23. Questions?
CONTACT
DANIEL GERLANC
ENPLUS ADVISORS, INC.
dgerlanc@enplusadvisors.com
https://github.com/dgerlanc/pyrtcbench