Performance engineeringforcloudcomputing lero

474 views

Published on

IDEAGEN Performance Engineering for Cloud Computing LERO

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
474
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
2
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Performance engineeringforcloudcomputing lero

  1. 1. Performance Engineering • Cloud overview Cloud overview for Cloud Computing • Previous results Previous results  • E Example: Logging  l L i John Murphy John Murphy Performance Engineering Lab • Future directions 8th European Performance Engineering Workshop ‐ EPEW 2011 Lero © 2011. Slide 1 8th European Performance Engineering Workshop ‐ EPEW 2011 Lero © 2011. Slide 2 Challenges in the Cloud The same What is Cloud Computing?approach(es) canbe applied tosolve queueingproblems in very XXXX as  a Service .... different areas While (Hype=True)  While (Hype=True) Can the same {approach(es) for Replace XXXX with Anything Replace XXXX with AnythingPerformanceEngineering be }applied to solveproblems in verydifferent areas Cloud Washing Required8th European Performance Engineering Workshop ‐ EPEW 2011 Lero © 2011. Slide 3 8th European Performance Engineering Workshop ‐ EPEW 2011 Lero © 2011. Slide 4 22 4
  2. 2. Cloud Users Evolving Critical Systems Cloud Computing “As‐A‐Service”  market sizing Report II, October  Report II, October 2010 Michael Armbrust, Armando Fox, Rean Griffith, Anthony D. Joseph, Randy Katz, Andy  Konwinski, Gunho Lee, David Patterson, Ariel Rabkin, Ion Stoica, and Matei Zaharia.  2010. A view of cloud computing. Commun. ACM 53, 4 (April 2010), 50‐58.  2010 A view of cloud computing Commun ACM 53 4 (April 2010) 50 588th European Performance Engineering Workshop ‐ EPEW 2011 Lero © 2011. Slide 5 22 5 8th European Performance Engineering Workshop ‐ EPEW 2011 Lero © 2011. Slide 6 22 6 Challenges in the Cloud Challenges in the Cloud Flavours of Cloud Computing So what’s really new: Public Cloud (Amazon, Google, Microsoft) • Infinite computing resources on demand Private Cloud (many) • No cap ex  Hybrid Cloud Computing • Pay for what you use Surge / Utility Computing Surge / Utility Computing8th European Performance Engineering Workshop ‐ EPEW 2011 Lero © 2011. Slide 7 22 7 8th European Performance Engineering Workshop ‐ EPEW 2011 Lero © 2011. Slide 8 22 8
  3. 3. Infinite computing resources on demand Infinite computing resources on demand Ability to follow surges in workload • No capacity planning required • Speed of surge important to provide capacity • Data center utilisation low • Lower price possible due to statistical multiplexing of  many demands (self‐similarity an issue) d d ( lf i il i i ) Workload varies over the day Workload varies over the season Workload varies over the season Workload varies with events8th European Performance Engineering Workshop ‐ EPEW 2011 Lero © 2011. Slide 9 22 9 8th European Performance Engineering Workshop ‐ EPEW 2011 22 Lero © 2011. Slide 10 10 Pay for what you use Challenges in the Cloud • Amazon: Physical Hardware (EC2 instances), control  Ten Challenges in Cloud Computing [1] kernel upwards, lots of state information 1. Business Continuity & Service Availability • Google: Applications on AppEngine (web applications),  2. Data Lock In separation between compute and storage 3. Data Confidentiality/Auditability • Microsoft: Azure more flexible than the AppEngine Mi ft A fl ibl th th A E i 4. Performance Unpredictability 4 P f U di t bilit 5. Scalable Storage Cost: 1 machine for 1000 hours = 1000 machines for 1 hour Cost: 1 machine for 1000 hours = 1000 machines for 1 hour 6. Bugs in Large Scale Distributed Systems 6 Bugs in Large Scale Distributed Systems 7. Scaling Quickly Pay as you go, or usage based pricing (not renting) y y g , g p g( g) 8. Reputation Fate Sharing Pay per box, or pay per resources used 9. Data Transfer Bottlenecks 10.Software Licensing [1] ”A View of Cloud Computing”, by Michael Armbrust, Armando Fox, Rean Griffith,  Anthony D. Joseph, Randy Katz, Andy  Konwinski, Gunho Lee, David Patterson,  Ariel Rabkin, Ion Stoica, and Matei Zaharia 22 228th European Performance Engineering Workshop ‐ EPEW 2011 Lero © 2011. Slide 11 11 8th European Performance Engineering Workshop ‐ EPEW 2011 Lero © 2011. Slide 12 12
  4. 4. Typical Enterprise Systems Data Data Everywhere • 150 billion GB (exabytes) of data created in 2005;  Eight  times that amount (1,200 exabytes) in 2010 ( , y ) • The amount of enterprise data will grow about 650% over  the next five years, the vast majority of it unstructured, or  not included in any database.  • Log data is the fasted‐growing data source at large  organizations  • Many organizations are currently producing terabytes of  log data per month  log data per month8th European Performance Engineering Workshop ‐ EPEW 2011 Lero © 2011. Slide 13 8th European Performance Engineering Workshop ‐ EPEW 2011 Lero © 2011. Slide 14 Cloud Services Log Management • Not deployed in house Not deployed in house • Automatic Collection, Analysis & Visualization of Log Automatic Collection, Analysis & Visualization of Log  Data • Services need to handle 1000’s of customers Services need to handle 1000 s of customers • Use Cases: • Services need to handle 1000’s of enterprise systems Services need to handle 1000 s of enterprise systems Problem Determination Problem Determination Operations • Processing higher volumes of data required Processing higher volumes of data required Security Compliance & Auditing8th European Performance Engineering Workshop ‐ EPEW 2011 Lero © 2011. Slide 15 8th European Performance Engineering Workshop ‐ EPEW 2011 Lero © 2011. Slide 16
  5. 5. Log Maths Typical Log Volumes Customer Type Log Volumes Events per Second Events per Day Large Cloud Provider 50 Terabyes per Day 2,000,000 172,000,000,000 100,000 log messages / second  100,000 log messages / second Large Social Media 25 Terabytes per Day 1,000,000 1 000 000 Organisation x 300 bytes / log message = 28.6 MB Telecom Middleware/ 1 Terabyte per Day 50,000 Applications x 3600 seconds   100 6 GB / hour x 3600 seconds ~ 100.6 GB / hour Large Organisation 300 GB Per Day 15,000 x 24 hours ~ 2.35 TB / day (>1000 employees) Online Marketing Org 100 GB per day 5,000 432,000,000 x 365 days  860 5 TB / year x 365 days ~ 860.5 TB / year Small 10 GBs per Day 500 Data Centre x 3 years ~ 2.52 PB SAAS Educational Tools 5Gbs Per Day 250 Single IBM Test Team 2 GBs per Day 100 Online Multimedia 700Mbs Per Day 35 From Anton Chuvakin’s Blog Aug 2010 http://chuvakin.blogspot.com/ Early Stage Start up E l St St t 50Mbs Per Day 50Mb P D 25 2,000,000 2 000 0008th European Performance Engineering Workshop ‐ EPEW 2011 Lero © 2011. Slide 17 8th European Performance Engineering Workshop ‐ EPEW 2011 Lero © 2011. Slide 18 Partial results in log management Real Time Correlation Engine RTCE • High volume data processing • IBM & UCD Research (since 2007) IBM & UCD Research (since 2007) • Correlation • In house deployment In house deployment • Searching / Indexing • In use across 10’s of IBM teams (Dublin US China) In use across 10 s of IBM teams (Dublin, US, China) • Pattern detection (symptom database) • Ability to process 80 000 events per second Ability to process 80,000 events per second • Real time requirements q8th European Performance Engineering Workshop ‐ EPEW 2011 Lero © 2011. Slide 19 8th European Performance Engineering Workshop ‐ EPEW 2011 Lero © 2011. Slide 20
  6. 6. RTCE details logentries.com Network of Componenta Nodea agents Log Nodeb Nodec Agent • Log Management as a Service Log Management as a Service Log Built on Amazon Web services Noded Componentb Testing environment Scales Horizontally Node detail Inter-agent communication o CPU  Presentation o Storage St Web server Agent Distributed File System/ NoSQL DBs (Hadoop) Distributed File System/ NoSQL DBs (Hadoop) Needs to handle TB per day (2TB per customer) p y( p ) Usera Userb Userc Userd8th European Performance Engineering Workshop ‐ EPEW 2011 Lero © 2011. Slide 21 8th European Performance Engineering Workshop ‐ EPEW 2011 Lero © 2011. Slide 22 21 Key log research challenges Conclusions • Cloud computing is a tag for the next while... • Scalable hardware resources Cloud  Auto scaling • Major issues still to be fully addressed • Indexing large volumes of data in real time • Previous performance engineering in enterprise,  No SQL / Columnar Storage grid, data centre or mainframe research can  grid data centre or mainframe research can probably feed into the solutions • Processing millions of events per second Bloom Filters Thank you! 228th European Performance Engineering Workshop ‐ EPEW 2011 Lero © 2011. Slide 23 8th European Performance Engineering Workshop ‐ EPEW 2011 Lero © 2011. Slide 24 24

×