Your SlideShare is downloading. ×
Admission Control in Impala
Admission Control in Impala
Admission Control in Impala
Admission Control in Impala
Admission Control in Impala
Admission Control in Impala
Admission Control in Impala
Admission Control in Impala
Admission Control in Impala
Admission Control in Impala
Admission Control in Impala
Admission Control in Impala
Admission Control in Impala
Admission Control in Impala
Admission Control in Impala
Admission Control in Impala
Admission Control in Impala
Admission Control in Impala
Admission Control in Impala
Admission Control in Impala
Admission Control in Impala
Admission Control in Impala
Admission Control in Impala
Admission Control in Impala
Admission Control in Impala
Admission Control in Impala
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Admission Control in Impala

1,664

Published on

Published in: Software
0 Comments
10 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,664
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
0
Comments
0
Likes
10
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. 1 Admission Control in Impala Matthew Jacobs | @mattjacobs | mj@cloudera.com
  • 2. 2 ©2014 Cloudera, Inc. All rights reserved. • Too many concurrent queries -> oversubscription • All queries take more time • Application layer can throttle queries? • Not much you can do before Impala 1.3 What’s the Problem
  • 3. 3 ©2014 Cloudera, Inc. All rights reserved. • Add an admission control mechanism to Impala! • Throttle incoming requests • Queues requests when workload increases • Queued requests executed when resources available So what do we do?
  • 4. 4 ©2014 Cloudera, Inc. All rights reserved. • Yarn is a resource manager for Hadoop • Assumes jobs are composed of tasks, tasks can be restarted • Need to ask for all resources up front, resources “trickle in” • Non-trivial overhead: job creates “application master” (AM) • But cost is small compared to long batch jobs • Great for MR, things like MR • Not good for • Low-latency, high volume workloads • Gang scheduling, “parts of jobs” can’t be restarted What about Yarn?
  • 5. 5 ©2014 Cloudera, Inc. All rights reserved. • “Long Lived Application Master” • Long running AMs • Create fake requests to acquire necessary resources • Provides a “gang scheduling” abstraction, waits for all resources • Offers a resource expansion mechanism -> don’t need to ask for all up front • Offers a throttling mechanism • Caches Yarn containers -> lower latency • Looks like a square peg in a round hole… • To be fair, multi-level scheduling is a hard problem! Llama Bridges the Gap
  • 6. 6 ©2014 Cloudera, Inc. All rights reserved. • Good for Impala sharing resources with other frameworks • Good general purpose resource mgmt solution However: • Not everyone wants/needs to run Yarn and Llama • Still requires round-trips to a central server • Increases query latency • Unlikely to scale for highest latency/throughput requirements • Impala should have a fast, built in throttling mechanism Impala + Llama + Yarn?
  • 7. 7 ©2014 Cloudera, Inc. All rights reserved. • Throttle number of concurrent requests or memory • Fast • Decentralized • Works without Yarn/Llama • Works with CDH4/CDH5 Impala Admission Control
  • 8. 8 ©2014 Cloudera, Inc. All rights reserved. • Configure one or more resource “pools” • Max # concurrent queries, max memory, max queue size • Each Impalad capable of making admission decisions • No new single bottleneck/single point of failure • Incoming queries are executed, queued, or rejected • Queue if too many queries OR not enough memory • Reject if queue is full Design Overview
  • 9. 9 ©2014 Cloudera, Inc. All rights reserved. • Requests admitted or queued locally • Each Impalad keeps track of local state • # queries, pool memory, local queue size • Disseminates local stats via statestore -> global state • Uses cached global state in admission decisions • Decisions are fast; negligible impact on query latency • No single point of failure Localized Admission Decisions
  • 10. 10 ©2014 Cloudera, Inc. All rights reserved. • Using cached global state -> may “over-admit” • E.g. multiple impalads think 1 request can be admitted and admit before receiving updated state • Configured pool limits are “soft” limits • Fn(Submission rate, distribution across impalads) • Not a big problem in practice • May occasionally admit a few extra queries • Can increase statestore heartbeat frequency • Can add some buffer to configured pool limits Localized Admission Decisions (II)
  • 11. 11 ©2014 Cloudera, Inc. All rights reserved. • Max memory • Many workloads are limited by memory • Impalads kill queries when running out of memory, anyway • Max number of concurrent queries • Generic mechanism, not resource specific (e.g. memory) • Not as good if workload is heterogeneous • Queries may still be killed if impalads run out of memory Pool Limits
  • 12. 12 Memory Limits • Impalads track memory hierarchically • Per-process memory • Queries killed when limit is hit • Per-pool memory • For admission control • Per-query memory Process Pool1 Query1 Query2 Pool2 ©2014 Cloudera, Inc. All rights reserved.
  • 13. 13 ©2014 Cloudera, Inc. All rights reserved. • Admission decisions need more than memory usage • Incoming queries use no memory yet • Queries recently admitted haven’t ramped up yet • Use memory estimates from planning • Estimate pool memory usage with actual usage & estimates • Accounts for future memory usage of recently started queries Admit if: Pool mem estimate + query mem estimate < pool limit Memory Limits (II)
  • 14. 14 ©2014 Cloudera, Inc. All rights reserved. • Not perfect, query mem estimates are wrong • Hard problem; never have perfect estimates from planning • Usually overly conservative • Leads to underutilization • But at least queries won’t be killed • Less likely to hit process mem limit • Workarounds • Increase pool mem limit • Override with “MEM_LIMIT” query option • Future improvement: Update estimates as query executes • Query mem usage will approach the updated estimate Memory Limits (III)
  • 15. 15 ©2014 Cloudera, Inc. All rights reserved. • Modeled after Yarn resource queues • Same configuration as Yarn queues • Have a single configuration for Yarn & Impala • Usually want to have the same resource allocations mapped to an organization • E.g. HR gets 10%, Finance gets 30%, Eng gets 60% Request Pools
  • 16. 16 ©2014 Cloudera, Inc. All rights reserved. • Users are mapped to pools using the placement policy • Users are authorized using the specified ACLs • Pools are defined hierarchically • ACLs are inherited • Currently only enforces limits on leaf pools (IMPALA-905) Request Pools (II)
  • 17. 17 ©2014 Cloudera, Inc. All rights reserved. • Uses Yarn + Llama configs • Yarn fair scheduler allocation configuration (fair- scheduler.xml) • Llama configuration (llama-site.xml) • Only some of the configuration properties are used • See the documentation for sample config files • Cloudera Manager has a nice UI to configure • No need to touch the xml files Request Pool General Configuration
  • 18. 18 ©2014 Cloudera, Inc. All rights reserved.
  • 19. 19 ©2014 Cloudera, Inc. All rights reserved. Placement Rule Configuration Please change the default values
  • 20. 20 ©2014 Cloudera, Inc. All rights reserved. • If only 1 pool is needed, a separate (easy) configuration path exists • Uses a single “default” pool • No Yarn/Llama configs involved (not even accepted) • Configure the pool limits with impalad flags: • default_pool_max_queued • default_pool_max_requests • default_pool_mem_limit • Doesn’t work with CM5.0, fixed in CM5.0.1 Easy Config Path (Singleton Pool Only)
  • 21. 21 ©2014 Cloudera, Inc. All rights reserved. Submitting to a Pool
  • 22. 22 ©2014 Cloudera, Inc. All rights reserved. • Rejections and timeouts return error messages • Metrics • Exposed in impalad web UI: /metrics • Will be available in CM5.1 • Query profile has admission result • Impalad logs have lots of useful information “Debugging” Admission Control Decisions admission-controller.cc:259] Schedule for id=c541aae43af74ed1:afdec812127f8097 in pool_name=root.test/admin PoolConfig(max_requests=20 max_queued=50 mem_limit=-1.00 B) query cluster_mem_estimate=42.00 MB admission-controller.cc:265] Stats: pool=root.test/admin Total(num_running=20, num_queued=7, mem_usage=239.07 MB, mem_estimate=800.00 MB) Local(num_running=20, num_queued=7, mem_usage=239.07 MB, mem_estimate=800.00 MB) admission-controller.cc:303] Queuing, query id=c541aae43af74ed1:afdec812127f8097
  • 23. 23 ©2014 Cloudera, Inc. All rights reserved. Metrics
  • 24. 24 ©2014 Cloudera, Inc. All rights reserved. Query Profile Information
  • 25. 25 ©2014 Cloudera, Inc. All rights reserved. • Queue timeout • Defaults to 60sec, change with --queue_wait_timeout_ms • Running with Yarn/Llama • Same configs: “hard limits” enforced by Yarn+Llama • Disabled by default for CDH4 • Hue (<CDH4.6) doesn’t close queries • Enable with impalad flag (see --disable_admission_control) Some Notes
  • 26. 26 ©2014 Cloudera, Inc. All rights reserved. Matthew Jacobs @mattjacobs mj@cloudera.com

×