Hasthi talk at ICWS 2009


Published on

You can find a detailed presentation of this under http://www.slideshare.net/hemapani/dissertation-defence-enforcing-userdefined-management-logic-in-large-scale-systems.

Published in: Education, Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • IT has become indispensable part of our life Millions of users => increase userbases => Big deal to Google or Amazon Successes depend on ability to make sense of data => Killer app of our time is Search, “Large Scale Search”
  • I said it is possible to build large scale systems, keeping them running is a different story!! Most of us have come to contact with large scale systems and no what it takes Changes->Operational cost->Unreliable Middleware Many solutions -> Quote Patterson -> System management as a potential solutions
  • Role of system management more like Human manager, watch over and control the system. We call the system being managed, a “managed system” I focus on Monitor -> Decide -> Execute NOT on specifications, rather how to use exposed data
  • This is outline
  • Resources, Managers, a special manager (Coordinator) Elected Among them, Bootstrap nodes (the entry point, not shown in the figure) Heartbeats , Resources => Manager, Manager => Coordinator Coordinator failed? Manager Failed? Resource Failed?
  • k.
  • Hasthi talk at ICWS 2009

    1. 1. ENFORCING USER-DEFINED MANAGEMENT LOGIC IN LARGE SCALE SYSTEMS Srinath Perera, Dennis Gannon Indiana University, Bloomington
    2. 2. Outline <ul><li>Motivation & the Problem : Managing Large Scale Systems using User-defined Management logic </li></ul><ul><li>Related Work </li></ul><ul><li>Proposed Solution : Hasthi </li></ul><ul><li>Scalability Results </li></ul><ul><li>Contributions </li></ul>
    3. 3. Motivation: Large Scale systems <ul><li>IT is becoming a part of our everyday life </li></ul><ul><ul><li>Increases size of potential user bases of systems (Google, Facebook, Amazon …). </li></ul></ul><ul><ul><li>Information Avalanche. </li></ul></ul><ul><ul><li>National, Global scale data collection </li></ul></ul><ul><ul><li>Success in this setting is decided by our ability to make sense of this data – scale matters (Google!). </li></ul></ul><ul><li>Technological advances </li></ul><ul><ul><li>Connectivity , SOA, Complex systems possible. </li></ul></ul><ul><ul><li>Computing power everywhere (multicore, smart phones). </li></ul></ul><ul><ul><li>Cloud - Lower the barrier for scale. </li></ul></ul>We have the need and means to build large scale systems
    4. 4. Building them is Feasible, but Keeping them Running ?? <ul><li>Changes are a norm rather than an exception – “10,000 servers, each having MTTF of thousand days => 10 failures/day” [Jeff Dean]. </li></ul><ul><li>High Operational Cost - When a system scales up, complexity increases. </li></ul><ul><ul><li>More than 75% TCO (Total Cost of Ownership) based on Patterson et al. data. (Dominated by salaries.) </li></ul></ul><ul><ul><li>50% IT budget spent on recovering from failures [Ganek et al.] </li></ul></ul><ul><li>Unreliable Middleware - Grid reliability among all operations 55%-80% [Khalili et al.]. Then the success rate of a service or a workflow that has 6 grid operations is 0.26 !!! </li></ul><ul><li>Efforts to avoid failures have been unsuccessful – “ Not a problem to be solved, but a fact to cope with” [ Patterson] </li></ul>
    5. 5. System Management As a Potential Solution + Role of User Defined Management Logic
    6. 6. The Problem <ul><li>Large scale systems need many managers </li></ul><ul><ul><li>One manager does not scale nor robust </li></ul></ul><ul><li>Each manager has a Partial view of the system </li></ul><ul><ul><li>a subset of resources are assigned to each manager </li></ul></ul><ul><li>But a Global view is Preferred ( ease of authoring logic ) </li></ul><ul><ul><li>Logic that work on local data need emergent properties, and hard for user to author them. </li></ul></ul><ul><ul><li>We all think in terms of global properties, </li></ul></ul><ul><li>Example : “If the system does not have 5 message brokers, create new brokers and connect them to the broker network.” : detect <5 brokers, find the best place to create new one, create new one, and connect it to existing brokers. </li></ul><ul><li>Problem: Enforcing user-defined management logic (that depends on a global view) on large-scale systems? And Application of such a framework to manage systems. </li></ul>
    7. 7. Approach Scalable Robust Ease of Writing management logic Problems Decentralized control (e.g. DMonA , and Deugo et al .) Highly Yes Hard Hard for users to write rules to achieve emergent behavior Complex Event processing (DREAM) Yes Possible Not Easy Event model has limited Memory Consistent view across managers (e.g. Georgiadis et al. ) No Yes Yes Need ordered reliable multicast – does not scale Hierarchical control with aggregation (Monalisa) Highly Possible Not Easy Lose identity of a single resource due to aggregation Hierarchy with Policies at each level (e. g. WildCat) Yes Possible Possible Policies are not as explicit as rules. State Machine (Dubey et al. ) Yes Possible Not Easy Users have to construct this state machine, which is hard
    8. 8. Our Solution <ul><li>Hasthi System Management Framework as a Solution </li></ul><ul><li>A Dynamic, Robust System Management Framework </li></ul><ul><li>We showed (details in next slides) that it can scale while enforcing user defined management logic that depend on Global assertions. </li></ul>
    9. 9. Big Picture (Hasthi) <ul><li>Hasthi Has three Parts </li></ul><ul><ul><li>Manager Cloud – distributed architecture that binds managers and resources in the system as one cohesive unit. </li></ul></ul><ul><ul><li>Meta-Model that represents the system state. </li></ul></ul><ul><ul><li>Decision Framework. </li></ul></ul>
    10. 10. Manager Cloud <ul><li>Managers form a P2P network (Pastry), which is used for Initialization and Recovery (Elections). </li></ul><ul><li>Normal Operations use SOAP over HTTP </li></ul>
    11. 11. Meta-Model <ul><li>Meta-model represents the monitoring data collected from the system. Summarized meta-model provides a global view. </li></ul><ul><li>Delta-consistency – changes are reflected within a bounded time (a concept borrowed from shared memory multiprocessors [see Singla et al. ]). </li></ul>
    12. 12. Decision Framework <ul><li>Users define management logic as rules: Local and Global. </li></ul><ul><li>Manager control loops evaluate partial meta-models using local rules. </li></ul><ul><li>The coordinator control loop evaluates the summarized meta-models using global rules (Global view). </li></ul><ul><li>Actions triggered by rules analyze meta-model and decide on solutions . </li></ul>
    13. 13. Management Rules <ul><li>Rules (Drools) evaluate meta-objects (which represent resources) and execute actions, which analyze meta-objects and decide on solutions. </li></ul>rule &quot;RestartFailedServices&quot; when service:ManagedService(state == &quot;CrashedState&quot;); host:Host(state != &quot;CrashedState&quot;, service.host == name); then system.invoke(new RestartAction(service), new ActionCallback() { public void actionSucessful(ManagementAction action) { ..... } public void actionFailed(ManagementAction action,Throwable e) { service.setState(&quot;UnRepairableState&quot;); system.invoke( new UserInteractionAction(system, service, action,e)); }}); end <ul><li>When the condition given using the object query language is met, actions in the then-clause are carried out. </li></ul><ul><li>Use Rete algorithm to evaluate meta-objects and execute corrective actions. Tradeoff between space and time . </li></ul>
    14. 14. Management Actions <ul><li>Action Types </li></ul><ul><ul><li>Create a New service </li></ul></ul><ul><ul><li>Restart a running service or recover a failed service </li></ul></ul><ul><ul><li>Relocate a service </li></ul></ul><ul><ul><li>Tune and configure a resource – change the configuration of a resource or change the structure of the system. </li></ul></ul><ul><ul><li>User Interaction Action </li></ul></ul><ul><li>Actions implementation: </li></ul><ul><ul><li>Use shell scripts (e.g. service start or stop) and execute them using a Host Agent running in each host. </li></ul></ul><ul><ul><li>Use Hasthi Agent integrated with each resource. </li></ul></ul><ul><li>Hasthi provides default management actions, but users can write their own. </li></ul>
    15. 15. Scalability: Test Setup <ul><li>Main Test Setup </li></ul><ul><li>Large scale deployment of LEAD. </li></ul><ul><li>Multiple replicas of the complete LEAD stack. </li></ul><ul><li>Each service simulates a management workload using a randomized algorithm. </li></ul><ul><li>Set of rules to manage the system, and each test ran for a 1 hour with 30 seconds epoch time. </li></ul><ul><li>Coordinator Test Setup: </li></ul><ul><li>Test-Manager that simulates all messages generated by a normal manager managing a set of resources. </li></ul><ul><li>We simulated a large-scale system using Test-Managers . </li></ul><ul><li>The coordinator does not see a difference. </li></ul>Q?
    16. 16. Measurements (Metrics)
    17. 17. One Manager Overhead (Resource Heartbeat Latency, Manager Loop Overhead, Manager Heartbeat Latency) Managers Overhead (Coordinator Loop, Manager Heartbeat ) <ul><li>One manager scales to 5000-8000 resources, Hasthi scales more with added managers. Need more tests to find the limits. </li></ul>
    18. 18. Scalability: Test Setup <ul><li>Main Test Setup </li></ul><ul><li>Large scale deployment of LEAD. </li></ul><ul><li>Multiple replicas of the complete LEAD stack. </li></ul><ul><li>Each service simulates a management workload using a randomized algorithm. </li></ul><ul><li>Set of rules to manage the system, and each test ran for a 1 hour with 30 seconds epoch time. </li></ul><ul><li>Coordinator Test Setup: </li></ul><ul><li>Test-Manager that simulates all messages generated by a normal manager managing a set of resources. </li></ul><ul><li>We simulated a large-scale system using Test-Managers . </li></ul><ul><li>The coordinator does not see a difference. </li></ul>Q?
    19. 19. Coordinator Limit: (Manager Heartbeat Latency, Coordinator Loop Overhead) vs. Resource count <ul><li>Close to a Linear overhead, the coordinator scales to 100,000 resources and 1000 managers, and the number of managers does not make a much difference. </li></ul><ul><li>Why? (1) Summarization , (2) Only transfer Changes , (3) Rete Algorithm , which only evaluates changes (tradeoff between speed vs. memory). </li></ul>
    20. 20. Manager Independence: (Resource heartbeat, Manager Loop vs. Manager Heartbeat) vs. resources per Manager <ul><li>We measured the limit of a manager and the limit of the coordinator. </li></ul><ul><li>Hypothesis: a manager overhead only depends on resources assigned to a manager, not on other manager s or resources in the system </li></ul><ul><ul><li>we can scale up Hasthi (e.g. 100 managers, 1000 resources each). </li></ul></ul><ul><li>Verify Hypothesis: </li></ul><ul><ul><li>A Scatter Plot: overhead vs. number of resources per Manager . </li></ul></ul><ul><ul><li>Same X values are reasonably close to each other. </li></ul></ul><ul><ul><li>Hypothesis is valid till 2000 resources at least. </li></ul></ul><ul><li>Why? Managers do not usually interact with other manager s , but talk with the coordinator. </li></ul>
    21. 21. Scalability: Summary <ul><li>One manager scales to 5000-8000 resources. </li></ul><ul><li>Managers only depend on resources assigned to them (at least till 2000 resources) and are not affected by other Manager s in the system. </li></ul><ul><li>Coordinator scales to 100,000 resources and 1000 managers (100-1000 resources per manager < 2000 limit in #2). </li></ul><ul><li>System scales to 100,000 resources. </li></ul>Q?
    22. 22. Sensitivity: Rules <ul><li>To find sensitivity to rules, 7 Rules sets, each having more rules then the one before, with 40,000 resources </li></ul><ul><li>Almost linear Overhead, seem to be stable. We also verified by running 100,000 resources against the most complex rule set. </li></ul>
    23. 23. Sensitivity: Epoch Time <ul><li>Epoch times are time periods between heartbeats and control loop evaluations etc, and they decide how fast Hasthi reacts to failures. </li></ul><ul><li>Why overhead reduce with smaller epoch? Rete algorithm remembers old results and only evaluates new results. Small epoch means less changes, which means less overhead!! </li></ul>
    24. 24. Sensitivity: Workload <ul><li>Increase failures in the system (increase workload on Hasthi) and measure with 40,000 resources. </li></ul><ul><li>Hasthi is stable, why? Hasthi uses a job queue to execute actions asynchronously. Therefore, can withstand higher workloads and surges. </li></ul>
    25. 25. Contributions <ul><li>Problem: Enforcing user-defined management logic (that depend on a global view of the managed system) on large-scale systems? </li></ul><ul><ul><ul><li>We proposed an architecture to solve this problem and demonstrated that despite its dependency on a global view, a Management Framework can scale to manage most real world usecases </li></ul></ul></ul>
    26. 26. Questions