Downloaded 10 times
Uploaded byKohei KaiGai
pgconfasia2016 lt ssd2gpu
Slides for LT of PGconf.ASIA2016 SSD-to-GPU P2P Transfer
More Related Content
![GPUとSSDがPostgreSQLを加速する～クエリ処理スループット10GB/sへの挑戦～ [DB Tech Showcase Tokyo/2017]](https://cdn.slidesharecdn.com/ss_thumbnails/20170906dbtsgpussdacceleratespostgresqljp-170906073226-thumbnail.jpg?width=640&height=640&fit=bounds)
![[20170922 Sapporo Tech Bar] 地図用データを高速処理！オープンソースGPUデータベースMapDってどんなもの？？ by 株式会社...](https://cdn.slidesharecdn.com/ss_thumbnails/20170922mapd-170926064811-thumbnail.jpg?width=640&height=640&fit=bounds)
![[db tech showcase Tokyo 2016] D13: NVMeフラッシュストレージを用いた高性能高拡張高可用なデータベースシステムの実現方...](https://cdn.slidesharecdn.com/ss_thumbnails/efmcwyyftdwdehhr3b4y-signature-06a70abedb3d16e5e7bcb9b2ba8014302148552ca76b90497742e559ab641b61-poli-160726094411-thumbnail.jpg?width=640&height=640&fit=bounds)
![[db tech showcase Tokyo 2015] B15:最新PostgreSQLはパフォーマンスが飛躍的に向上する！？ - PostgreSQ...](https://cdn.slidesharecdn.com/ss_thumbnails/dbts-tokyo-2015b15postgresqlntt-oss-center-150619073139-lva1-app6891-thumbnail.jpg?width=640&height=640&fit=bounds)
pgconfasia2016 lt ssd2gpu
- 1. Performing SQL withSSD-to-GPU P2P Transfer かぴばらの旦那 / Herr.Wasserschwein <kaigai@kaigai.gr.jp>
- 2. The PG-Strom Project Feedbacksunder the PG-Strom v1.0 development PGconf.ASIA2017 - LT / Performing SQL with SSD-to-GPU P2P Transfer2 Application Storage Query Optimizer Query Executor PG-Strom Extension SQL Parser Storage Manager GPU 計算集約的ワークロード computing intensive workloads • 統計解析、科学技術計算、 マーケティング、etc... • statistics, science, marketing, ... by PL/CUDA + Matrix-Array I/O集約的ワークロード (i/o intensive workloads) • DWH, ETL, Reporting, ... by SSD-to-GPU P2P DMA
- 3. The PG-Strom Project x86サーバのアーキテクチャ/ Architecture of x86 server PGconf.ASIA2017 - LT / Performing SQL with SSD-to-GPU P2P Transfer3 RAMRAM PCI Bus NVMe-SSD GPU PCIe x16PCIe x4~x8 PCH その他低速デバイス (other slow devices)
- 4. The PG-Strom Project x86サーバのアーキテクチャ/ Architecture of x86 server PGconf.ASIA2017 - LT / Performing SQL with SSD-to-GPU P2P Transfer4 PCH RAMRAM PCI Bus NVMe-SSD GPU PCIe x16PCIe x4~x8 I/O READ disk block disk buffer カタログスペック (catalog spec) 2GB~6GB/s その他低速デバイス (other slow devices)
- 5. The PG-Strom Project x86サーバのアーキテクチャ/ Architecture of x86 server PGconf.ASIA2017 - LT / Performing SQL with SSD-to-GPU P2P Transfer5 PCH RAMRAM PCI Bus NVMe-SSD GPU PCIe x16PCIe x4~x8 disk buffer その他低速デバイス (other slow devices)
- 6. The PG-Strom Project やりたい事/ What I want to do PGconf.ASIA2017 - LT / Performing SQL with SSD-to-GPU P2P Transfer6 PCH RAMRAM PCI Bus NVMe-SSD GPU PCIe x16PCIe x4~x8 result buffer disk block その他低速デバイス (other slow devices)
- 7. The PG-Strom Project やりたい事/ What I want to do PGconf.ASIA2017 - LT / Performing SQL with SSD-to-GPU P2P Transfer7 PCH RAM PCI Bus NVMe-SSD GPU PCIe x16PCIe x4~x8 Large PostgreSQL Tables Small Inner Tables WHERE句 JOIN GROUP BY データサイズが すごく小さく！ (making data-size much smaller!) その他低速デバイス (other slow devices)
- 8. The PG-Strom Project やりたい事/ What I want to do PGconf.ASIA2017 - LT / Performing SQL with SSD-to-GPU P2P Transfer8 PCH RAM PCI Bus NVMe-SSD GPU PCIe x16PCIe x4~x8 Large PostgreSQL Tables Small Inner Tables WHERE句 JOIN GROUP BY ここまで完成 (works completed) その他低速デバイス (other slow devices)
- 9. The PG-Strom Project 要素技術/ Element Technology： GPUDirect RDMA by NVIDIA PGconf.ASIA2017 - LT / Performing SQL with SSD-to-GPU P2P Transfer9 GPUのデバイスメモリを、物理アドレス空間 にマップするためのAPI (API to map GPU’s device memory on physical address space of the host system) ストレージからのDMA転送先に GPU上のデバイスメモリを指定できる。 (GPU’s device memory can be used for the destination address of DMA from the storage)
- 10. The PG-Strom Project NVMe-Stromドライバ/ NVMe-Strom Driver PGconf.ASIA2017 - LT / Performing SQL with SSD-to-GPU P2P Transfer10 pg-strom NVMe-Strom VFS Page Cache NVMe SSD Driver nvidia driver PostgreSQL /proc/nvme-strom read(2) User Space Kernel Space
- 11. The PG-Strom Project NVMe-Stromドライバ/ NVMe-Strom Driver PGconf.ASIA2017 - LT / Performing SQL with SSD-to-GPU P2P Transfer11 pg-strom NVMe-Strom VFS Page Cache NVMe SSD Driver nvidia driver GPU device memory PostgreSQL cuMemAlloc() /proc/nvme-strom read(2) User Space Kernel Space
- 12. The PG-Strom Project NVMe-Stromドライバ/ NVMe-Strom Driver PGconf.ASIA2017 - LT / Performing SQL with SSD-to-GPU P2P Transfer12 pg-strom NVMe-Strom VFS Page Cache NVMe SSD Driver nvidia driver GPU device memory GPU device memory PostgreSQL cuMemAlloc() /proc/nvme-strom ioctl(2) read(2) User Space Kernel Space
- 13. The PG-Strom Project NVMe-Stromドライバ/ NVMe-Strom Driver PGconf.ASIA2017 - LT / Performing SQL with SSD-to-GPU P2P Transfer13 pg-strom NVMe-Strom VFS Page Cache NVMe SSD Driver nvidia driver GPU device memory GPU device memory PostgreSQL file offset block number cuMemAlloc() /proc/nvme-strom ioctl(2) read(2) User Space Kernel Space
- 14. The PG-Strom Project NVMe-Stromドライバ/ NVMe-Strom Driver PGconf.ASIA2017 - LT / Performing SQL with SSD-to-GPU P2P Transfer14 pg-strom NVMe-Strom VFS Page Cache NVMe SSD Driver nvidia driver GPU device memory GPU device memory PostgreSQL file offset DMA request block number cuMemAlloc() /proc/nvme-strom ioctl(2) read(2) User Space Kernel Space
- 15. The PG-Strom Project NVMe-Stromドライバ/ NVMe-Strom Driver PGconf.ASIA2017 - LT / Performing SQL with SSD-to-GPU P2P Transfer15 pg-strom NVMe-Strom VFS Page Cache NVMe SSD Driver nvidia driver GPU device memory GPU device memory PostgreSQL file offset DMA request block number SSD-to-GPU Peer-to-Peer DMA cuMemAlloc() /proc/nvme-strom ioctl(2) read(2) User Space Kernel Space
- 16. The PG-Strom Project 単純I/O性能/ Raw I/O Performance PGconf.ASIA2017 - LT / Performing SQL with SSD-to-GPU P2P Transfer16 32MB x 6個のバッファを使用。バッファが空になる度に非同期DMAをキック 6 of 32MB buffers were used. Async DMA was kicked per 測定環境 / Environment CPU: Xeon E5-2670 v3, RAM: 64GB Intel SSD 750 (400GB; PCI-E x4) NVIDIA Tesla K20c (2496core; 706MHz, 5GB GDDR5; 208GB/s) OS: CentOS 7 (3.10.0-327.18.2.el7.x86_64), Filesystem: Ext4 カタログスペック！ (catalog spec!!)
- 17. The PG-Strom Project 測定に使用したNVMe-SSD/ NVMe-SSD for this measurement PGconf.ASIA2017 - LT / Performing SQL with SSD-to-GPU P2P Transfer17 容量 (capacity) 順次128KB 読出し (Seq Read) 順次128KB 書込み (Seq Write) ランダム4KB 読出し (Random Read) ランダム4KB 書込み (Random Write) インターフェース (Interface) 400GB 2,200MB/s 900MB/s 430,000 IOPS 230,000 IOPS PCIe 3.0 x4 800GB 2,100MB/s 800MB/s 420,000 IOPS 210,000 IOPS PCIe 3.0 x4 1.2TB 2,500MB/s 1,200MB/s 460,000 IOPS 290,000 IOPS PCIe 3.0 x4 これ以外に、Samsung PM1725 NVMe SSD (1.6TB, 6GB/s) での動作報告あり。 (working at Samsung PM1725 NVMe SSD (1.6TB, 6GB/s) was reported) Raw-I/OのSSD-to-GPUで5634MB/sを記録との報告 (It said the raw-I/O SSD-to-GPU worked with 5634MB/s) https://github.com/kaigai/nvme-kmod/issues/1
- 18. The PG-Strom Project SQLスキャン性能/ SQL Scan Performance PGconf.ASIA2017 - LT / Performing SQL with SSD-to-GPU P2P Transfer18 ▌この測定結果から分かる事 / What this measurement tells us 既存ストレージ層の性能限界 / Performance limit of the storage layer 64GB / 140sec = 468MB/s Raw-I/O性能(587MB/s)に20%程度の追加コスト (Extra 20% cost in addition to the raw-i/o throughput (587MB/s)) NVMe-Stromによる改善 / Improvement by NVMe-Strom スループット / Throughput： 64GB / 43sec = 1524MB/s Existing Limit
- 19. The PG-Strom Project 測定に使用したクエリ/ Query for the measurement PGconf.ASIA2017 - LT / Performing SQL with SSD-to-GPU P2P Transfer19 CREATE TABLE t_64g (id int not null, x float not null, y float not null, z float not null, memo text); INSERT INTO t_64g (SELECT x, random()*1000, random()*1000, random()*1000, md5(x::text) FROM generate_series(1,700000000) x); postgres=# ¥d+ List of relations Schema | Name | Type | Owner | Size | Description --------+------------------+-------+--------+---------+------------- public | t | table | kaigai | 965 MB | public | t_64g | table | kaigai | 66 GB | Query-1) Scan query with a simple WHERE-clause SELECT * FROM t WHERE x BETWEEN y-2 AND y+2; Query-2) Scan query with a complicated WHERE-clause SELECT * FROM t_64g WHERE sqrt((x-200)^2 + (y-300)^2 + (z-400)^2) < 10; Query-3) Scan query with text matching SELECT * FROM t WHERE memo LIKE '%abcd%';
- 20. The PG-Strom Project 開発ロードマップ/ Development Roadmap PGconf.ASIA2017 - LT / Performing SQL with SSD-to-GPU P2P Transfer20 • GPUデバイスメモリのホストマッピングと、SSD-to-GPU P2P DMA要求の発行 • Host mapping of GPU device memory, and P2P DMA request for SSD-to-GPU transfer ① NVMe-Strom driver: the basic functionality • PostgreSQL v9.6のCPU並列対応と、NVMe-Stromを使ったP2Pのデータロード • hybrid parallel, and peer-to-peer data loading by NVMe-StromSupport of CPU+GPU ② PG-Strom: Integration with GpuScan + NVMe-Strom • PostgreSQL v9.6の新オプティマイザ対応と、スキャン実装のGpuScanとの統合 • Support of new optimizer in PostgreSQL v9.6, and integration with GpuScan for simple scan ③ PG-Strom: JOIN/GROUP BY Support • RAID-0/1区画に対するストライピングREAD / Striping READ on RAID-0/1 volumes ④ NVMe-Strom driver: RAID-0/1 support • テスト、テスト、テスト、デバッグ / Test, Test, Test, Debug ⑤ 品質改善・安定化 / Quality improvement and stabilization ⑥ PG-Strom v2.0!! （2017/2Q～3Q） いまココ！！
- 21. The PG-Strom Project PG-Stromv2.0のターゲット / Target on PG-Strom v2.0 PGconf.ASIA2017 - LT / Performing SQL with SSD-to-GPU P2P Transfer21 PCI-E x8 5.0GB/s PCI-E PCI-E x16 ~10GB/s シングルノードで最大20GB/sのデータ処理能力を目指す (towards 20GB/s data processing capability per node) Dual NVMe-SSD +RAID0/1対応 10GB/sのスループットで SSDブロックをGPUへロード 数千コアによる GPU並列処理 PCI-E PCI-E x8 5.0GB/s PCI-E x8 5.0GB/s PCI-E x16 ~10GB/s PCI-E x8 5.0GB/s Dual NVMe-SSD +RAID0/1 Support Loading SSD blocks to GPU with 10GB/s throughput GPU Parallels by thousands cores
- 22.