Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Presto in my_use_case2

1,270 views

Published on

presto

Published in: Data & Analytics
  • Be the first to comment

  • Be the first to like this

Presto in my_use_case2

  1. 1. Presto in my use case 2 Presto Meetup 2016/07/19 http://eventdots.jp/event/593112 @wyukawa
  2. 2. About 1.5 year ago • I talked about “Presto in my use case” at Presto Meetup(http://eventdots.jp/event/276987) on 2015/01/20 • http://www.slideshare.net/wyukawa/presto- in-myusecase/
  3. 3. Log Analysis System(2015/01/20) Hadoop, Hive of HDP2.1 Azkaban 2.6.4 Presto 0.89 Cognos 10.2.1 Prestogres 0.4.8 DB MySQL 5.5 DBDB ETL with Python 2.7.7 InfiniDB Pentaho Saiku Shib
  4. 4. Log Analysis System(2016/07/19) Hadoop, Hive of HDP2.1 Azkaban 3.0.0 Presto 0.147 Cognos 10.2.2 Prestogres 0.6.7 DB Netezza DBDB ETL with Python 2.7.11 InfiniDB Pentaho Saiku yanagishima
  5. 5. How do I use Presto? • batch with Hive on MR2(not Tez) • select only with Presto – execute adhoc query – report by Cognos with Prestogres • data planners create presto views
  6. 6. What is Prestogres? PostgreSQL pgpool-II (patched) BI tool Presto Prestogres PL/python
  7. 7. Toubles related to Prestogres • data type supported by Presto increases – varchar(0), tinyint • Postgresql sometimes doesn’t support some data type(varchar(0), tinyint) • https://github.com/treasure- data/prestogres/issues/46 • https://github.com/treasure- data/prestogres/pull/53
  8. 8. Presto Setting • -Xmx 16G • jdk1.8.0_92 – avoid query stuck at jdk1.8.0_45 – https://github.com/prestodb/presto/issues/5353 • query.max-memory=70GB • query.max-memory-per-node=3GB • task.cpu-timer-enabled=false • exchange.http-client.request-timeout=120s – avoid PAGE_TRANSPORT_TIMEOUT – https://github.com/prestodb/presto/issues/4292
  9. 9. Upgrade Presto • frequent version up • easy to upgrede because of stateless, but sometimes need to rollback – 0.101 https://github.com/prestodb/presto/pull/2834 – 0.108 https://github.com/prestodb/presto/pull/3212 • query stuck • revert commit – 0.113 https://github.com/prestodb/presto/pull/3400 – 0.148 https://github.com/prestodb/presto/pull/5612 • memory error
  10. 10. yanagishima • yanagishima is the web application for Presto – yanagishima is place-name at Chigasaki • There are similar tools – airpal – shib – hue • but, there are some problems, so I decided to implement another tool. – https://github.com/wyukawa/yanagishima
  11. 11. screenshot
  12. 12. yanagishima feature • share query and result through parmanent link • kill query • create presto view • easy to see data corresponding to column

×