The document outlines a meeting agenda for the analytics infrastructure team at Lyft, focused on the usage and setup of Presto, eventing, monitoring, and query protection strategies. It details historical data handling, performance issues, and the implementation of a query replayer for diagnostics, along with suggestions for future work on improving the system and enhancing query routing. Ultimately, it serves as a comprehensive guide to address Presto query performance and monitoring challenges.

1. Nishant Rayan
Analytics Infra Team
Presto @ Lyft

3. Agenda

4. 4
History
Usage
Setup
Eventing & Monitoring
Replayer
Gateway
Query protection
Future work
Q&A
1
2
3
4
5
6
7
8
9

5. History

6. 6
History
Redshift
(Interactive &
Batch)
2013

7. 7
History
Redshift
2013
Redshift
(Interactive &
Batch)
Hive (Batch)
2016

8. 8
History
Redshift
2013
Redshift
Hive
2016
Redshift (Batch /
Interactive)
Hive (Batch)
Presto (Interactive)
2017

9. 9
History
Redshift
2013
Redshift
Hive
2016
Redshift
Hive
Presto
2017
Redshift (Batch)
Hive (Batch)
Presto (Interactive)
Druid (Interactive)
2018

10. Presto Setup

11. Setup

12. Setup

13. Setup

14. Setup

15. Setup

16. Setup

17. Setup

18. Setup

19. Setup
Presto version 0.198
Coordinator c5.18xlarge
Workers 120+ nodes, m5.12xlarge
Hive metastore 2.1
Metastore DB AWS RDS

20. Setup
Data
● Data store - S3 - >> 10 PB
● Parquet format
● Data Partitioned by date
● 50K+ tables

21. Usage

22. Traffic patterns
• Readonly
• Mostly Adhoc queries (95%)
• Peak during business hours
• Scheduled reports

23. Event Collection

24. Scenario
Why are my presto queries very slow today?

25. Causes
● Maxed out resources
● Bad query

26. Resource utilization
Maxed out resources -> High query queue time

27. Requirement
Graph that shows user’s query queue time

28. Data
• User / Source
• Query times (received, execution start time, end time)
• Query text
• Query failure info (type, exception)

29. Event Collection
Mechanism
• Leverage presto plugin
• Collect rich presto query events
• Sent to s3 as json
• Generate usage / failure report

30. User Query Performance

31. User experience dashboard

32. More use cases...
• Most popular presto query tool (mode vs looker vs superset vs cmd line)
• Queries affected during downtime and cluster restarts
• Most popular table users like to query using presto
• Most expensive query
• Top users of presto

33. Monitoring
System Metrics
• Host level metrics (CPU, mem)
Cluster Metrics
• Active nodes
• Cluster memory
Query metrics
• Failures (internal, external, IRE)
• Active, finished query counts
• Beacon query status

34. Monitoring

35. Replayer

36. Scenario
A lot of presto queries timed out . What happened ? and How to fix it ?

37. Causes
● Long running queries
● Coordinator busy

38. Requirement
● Play queries in the exact same order
● Mimic actual users firing queries

39. Query Replayer

40. Query Replayer

41. Query Replayer

42. Query Replayer

43. Query Replayer

44. More use cases
● Shadow mode for upgrade confidence
● Perf testing resource group policies

45. Benchmarking
Other tools
• Presto – Verifier
• Compare perf of 2 clusters
• Upgrade version testing
• Benchto
• Run set of queries
• Save performance numbers and compare

46. Gateway

47. Scenario
Scheduled queries don’t require the same SLA as adhoc queries

48. Scenario
…. but they do affect adhoc query experience

49. Solution
Use separate cluster for adhoc and scheduled queries

50. Problem
But not all clients support transparent query routing

51. Requirement
Route queries based on rules
Route query status checks to right cluster

52. Gateway

53. Gateway

54. Gateway

55. Gateway

56. Gateway

57. Gateway

58. Gateway

59. Gateway

60. Gateway

61. Gateway

62. Query Protection

63. Scenario
User ran
“SELECT * FROM hive.raw.event_ride_happened”

64. Scenario
User ran
“SELECT * FROM hive.raw.event_ride_happened”

65. Scenario
User ran
“SELECT * FROM hive.raw.event_ride_happened WHERE ds = ‘2018-01-01’”

66. Problem
● Scans large volumes of data
● Starves other queries
● Times out

67. Requirement
● Stop bad queries early
● Pluggable rule set

68. Query Protection
● Leverage presto plugin
● Presto parser to extract info from query
● Apply heuristic rules to block
● Query on table without partition
● Query did not specify partition
● Non-standard query access (CLI etc..)
● Previously known bad queries

69. Gateway Replayer
Query
Protection
Single point of entry for
multiple presto clusters
Reproduce issues in
performance
Block bad queries
Takeaways

70. Future work

71. • Improve Gateway
• Caching
• Benchmark infra
• Presto K8s
• Open source replayer, audit framework, gateway

72. Q & A

73. Team

74. Thank You
eng.lyft.net
contact-nishant@lyft.com