This document summarizes how a media company stores and processes over 15 million assets totaling 3 petabytes of data using PostgreSQL instead of traditional file systems. The PostgreSQL database stores metadata, file hashes, and derivatives for ingestion, processing, distribution and analysis while handling over 200GB of core data and 500GB of XML processing. A custom solution was built using PostgreSQL data types, hashing functions, and asynchronous actions to support the large scale storage and processing needs of the company.

1. Dispatching Petabytes
with PostgreSQL
Andrew Pantyukhin
andrew@dreamindustries.ru

2. 15M media objects
3PB raw data
storage, streaming, processing

3. HDFS? Isilon?
custom solution

4. 1000s hard drives
file system per drive
filename = sha256(file)

5. dispatching
ingestion, rebalancing
encoding, analysis

6. PostgreSQL!
(of course)

7. entities
sha (asset), hdd, chassis
metadata, actions, status

8. 15M master objects
25M derivatives
70M copies

9. 200GB core
500GB XML processing
2TB+ overall

10. custom types
enum
native/wrappers

11. hashtypes
shatypes
+ crc32, bugfixes

12. actions
fully async, fail-over
dumb polling

13. smart locking
update set t=now()where t old
update returning

14. XML
third-party metadata
stored, processed in PG

15. research
large-scale action logging

16. production
aggregated views of dispatcher

17. distributed logic
dispatcher, XML processing,
production, research
full-mesh data exchange

18. table data transfer
slow or inflexible
simple custom scripts, diff

19. dream industries
disruptive innovation lab
funding, collaborating
inviting, hiring