Recommended
More Related Content
What's hot
What's hot (20)
Similar to Exasol User Group, 8 Oct 2018: PyEXASOL and UDF
Similar to Exasol User Group, 8 Oct 2018: PyEXASOL and UDF (20)
Recently uploaded
Recently uploaded (20)
Exasol User Group, 8 Oct 2018: PyEXASOL and UDF
- 1. Data ingestion with PyEXASOL and UDFs Vitaly Markov
- 2. What is Badoo? ❖ Dating-focused ❖ Social network ❖ Multiple brands ❖ ~ 190 countries ❖ ~ 47 languages ❖ ~ 400,000,000 users
- 3. Exasol setup ❖ Bare-metal hardware ❖ 20 (+1) nodes: ➢ ~ 480 CPU cores total ➢ ~ 14 Tb RAM total ➢ ~ 350 Tb raw hot data ❖ ~ 20,000,000,000 events daily ❖ ~ 98% cache hit rate
- 5. ❖ Network-bound (WiFi, VPN) ❖ Benefits from compression ❖ CPU-bound, but many cores ❖ Benefits from multiprocessing Person with laptop Server in datacenter
- 6. PyEXASOL ❖ Websocket protocol ❖ Python 3.6+ ❖ HTTP transport, compression ❖ Multiprocessing ❖ Feature-rich ❖ For analysts with laptops ❖ For parallel machine learning
- 7. What's wrong with ODBC? ODBC driver Python objects Pandas DataFrame slow
- 8. HTTP Transport Pandas DataFrame (C reader) ➢ compression ➢ encryption
- 9. Parallel HTTP Transport 200M users are scored in 3 minutes
- 10. 100% CPU usage on all cores Progress tracking:
- 11. How it looks in Exasol EXPORT my_table INTO CSV AT 'http://27.1.0.30:33601' FILE '000.csv' AT 'http://27.1.0.31:41733' FILE '001.csv' AT 'http://27.1.0.32:45014' FILE '002.csv' AT 'http://27.1.0.33:42071' FILE '003.csv' AT 'http://27.1.0.34:36669' FILE '004.csv' AT 'http://27.1.0.35:36794' FILE '005.csv'
- 12. Other features ❖ Multi-host connection strings ❖ SQL formatter ❖ SSL encryption ❖ Local config (.ini file) ❖ Metadata requests ❖ Profiling of last query ❖ Built-in UDF script output server
- 13. GitHub: https://github.com/badoo/pyexasol/ PyPI: https://pypi.org/project/pyexasol/ > pip install pyexasol Available now
- 14. Take a breather
- 15. UDF Framework ❖ Run anything in Exasol ❖ In massively parallel manner ❖ Very flexible ❖ Great for IMPORT / EXPORT ❖ But: ➢ Requires some skill ➢ Lack of practical manual
- 16. Parallel import from MySQL ❖ MariaDB JDBC driver ❖ Protocol compression ❖ ~ 200 MySQL hosts ❖ ~ 376,400 shards ❖ Up to 800 parallel threads ❖ 500,000,000 rows in 10m
- 17. Hadoop ORC support ❖ Columnar format ❖ Native Java API ❖ VectorizedRowBatch (fast!) ❖ Exasol can pre-sort data ❖ Optional bloom filters ❖ Many custom settings
- 18. Logging best practices INF VM:139966036683480 B:1 I:16 2018-09-22 18:30:57.718 [Start BATCH] {"total_memory":61,"db_host":"dbs156.mlan","max_memory":494} INF VM:139966036683480 B:1 I:16 2018-09-22 18:30:57.939 [Finish spot] {"spot_id":"189607","db_host":"dbs156.mlan","row_count":16} INF VM:139966036683480 B:1 I:16 2018-09-22 18:30:57.976 [Finish spot] {"spot_id":"189614","db_host":"dbs156.mlan","row_count":24} INF VM:139966036683480 B:1 I:16 2018-09-22 18:31:00.788 [Finish BATCH] {"total_memory":61,"total_duration":3.067,"row_count_total":2924,"spot_count_total ":174,"fetch_duration":0.695,"db_host":"dbs156.mlan","statement_duration":1.99,"co mpression":"ON","bytes_sent":"797967","max_memory":494} PyEXASOL function: execute_udf_output()
- 20. Memory pitfall Exasol DB RAM Cluster OS UDF Pre-allocated during startup Set in ExaOperations: DB RAM, Huge Pages Required for OS Tiny! ❖ Always monitor memory usage ❖ Exasol may lose parallelism if >4Gb RAM used per node ❖ Increase this limit if you need it (contact Customer Support)
- 21. Long transaction pitfall Process 2 Process 1 Process 3 Process 4 ❖ Some groups are slow ❖ Not obvious ❖ Hard to profile ❖ Holding locks ❖ Holding resources ❖ QUERY_TIMEOUT helps
- 22. Gists ❖ Exasol UDF: Import from MySQL ❖ Exasol UDF: Import from Hadoop ORC ❖ Exasol UDF: Export to Hadoop ORC Icons from the Noun Project Thank you!