Uploaded byWolfgang Grieskamp
Rv11
The document discusses runtime analysis challenges and opportunities in cloud computing, specifically focusing on monitoring and testing methodologies used by Google. It highlights distinct approaches such as black-box and white-box monitoring, the role of logs, and model-based testing, while addressing issues like state management and the integration of testing frameworks. Overall, the author emphasizes that while cloud computing introduces new operational paradigms, it doesn't fundamentally change the underlying problems in runtime verification and testing.
Rv11
- 1. Runtime Analysis inthe Cloud:Challenges and OpportunitiesWolfgang GrieskampSoftware Engineer, Google Corp
- 2. About Me < 2000:Formal Methods Research Europe (TU Berlin)2000-2006: Microsoft Research: Languages and Model-Based Testing Tools2007-2011: Microsoft Windows Interoperability Program: protocol testing and tools Since 4/2011: Google: Google+ platform and toolsDISCLAIMER: This talk does not necessarily represent Google’s opinion or direction.
- 3. Content of thistalkGeneral blah blah about cloud computingMonitoring the CloudTesting the CloudUsing the Cloud for DevelopmentA formal method guy’s dream…Conclusion
- 4.
- 5. What is CloudComputing?From Wikipedia, the free encyclopediaCloud computing is the delivery of computing as a service rather than a product, whereby shared resources, software and information are provided to computers and other devices as a utility (like the electricity grid) over a network (typically the Internet).
- 6. What is CloudComputing?From Wikipedia
- 7. Some Properties ofCloud ComputingIt is location independent
- 8. It is deviceindependent
- 9. It has aunified access control model
- 10. You pay foronly what you use
- 11. It can scale-upor scale-down on demand
- 12.
- 13.
- 14. IAAS (Infrastructure AsA Service)Basic building blocksStorageNetworkingComputationHomogenous, easy migration (based on VMs)Distributed Data Centers over Geographical ZonesPlayers: Amazon, GoGrid, Rackspace, Microsoft, GoogleEstimated revenue 2010: $1b (Source: Economist)
- 15. Platform As AService (PAAS)Basic Building BlocksOperating System Frameworks and Development ToolsDeployment and Monitoring ToolsPlayers: Microsoft, Google, IBM, SAP, …Estimated Revenue 2010: $311m (Source: Economist)
- 16. Software As AService (SAAS)Device and location independent applications, typically running in a browser (Email, Social, Retail, Enterprise Apps, etc.)Many different playersEstimated revenue 2010: $11.7b (Source: Economist)
- 17.
- 18. Monitoring vs RVvs TestingWhat’s the difference?A (strict) take:Monitoring collects and presents information for human analysisRuntime verification collects and transforms information for automated analysis which ultimately leads to a verdictTesting does the above things in an isolated, staged or mocked, environment. In particular, stimuli from the environment are simulated. In practice, boundaries are not so clear. For this talk RV = Monitoring (adapting to Google conventions)
- 19. Anatomy of aData CenterData Center AData Center B ……ControllerServerControllerServerServerServerServer…StorageStorageStorageStorageStorageNote: abstracted and simplified
- 20. Anatomy of aServerData Center AData Center B ……ControllerServer (VM)ControllerServerControllerJobJobJobServerServerServerServer…MonitorMonitorMonitorStorageStorageStorageStorageLogsAlertNote: abstracted and simplified
- 21. Anatomy of aServiceData Center AData Center B ……ControllerService (across Servers)JobJobServerControllerJobJobJobServerServerServerServer…JobStorageStorageStorageStorageStorageStorageNote: abstracted and simplified
- 22. Monitoring Types @GoogleBlack Box MonitoringWhite Box MonitoringLog Analysis
- 23. Black Box MonitoringJobMonitorFrequentlysend requests and analyze the response Possible because server jobs are ‘stateless’ and always input enabledIf failure rate over a certain time interval exceeds a given ratio, raise an alert and page an engineerEngineers aim for minimizing paging and avoiding false positives
- 24. Black Box Monitoring:How its done @ GoogleThere are rule based languages for defining request/responses. Each rule:Synthesizes an HTTP requestAnalyzes the response using a regular expressionSpecifies frequency and allowed failure ratioRules are like tests: a simple trigger and a simple response analysis Monitors can be also custom codeJobMonitor
- 25. Black Box Monitoring:Issues?Is the ‘stateless’ hypothesis feasible?Nothing is really stateless -- state is passed as parameters in cookies, continuation tokens, etc.However, as these are health tests, state can be ignoredWhat is the relation to testing?In theory very similar, only that the environment is not mocked. In practice uses quite different frameworks/languages What about service/system level monitoring?Its only about one job. Doesn’t give failure root cause (it only measures a symptom)JobMonitor
- 26. ChallengeIntegrate Black-Box Monitoringand TestingJobMonitorBlack-box monitoring can be seen as particular way of executing tests end-to-end on the live product such that the impact on performance can be neglected.Frameworks which integrate design and execution of monitoring rules and test cases are promisingMainly an engineering challenge
- 27. JobJobChallengeSystem/Service Level Black-BoxMonitoringMonitorMonitorNot commonly done Main purpose would be failure cause analysis and failure preventionSimple local monitoring already discovers failures Is there a strong point of doing it at runtime (vs. log analysis)?Only if real-time prevention and potentially repair is important Monitor
- 28. ChallengeProtocolcontract verificationJobMonitorAt Google,all communication between jobs happens via a single homogeneous RPC mechanism based on a message format definition language (called protocol buffers)Also all (= terra bytes) of data is stored in formats specified by protocol buffersOne could formulate data invariant and protocol sequencing contracts over protocol buffers and enforce them at runtime
- 29. White-Box MonitoringServer exportscollection of probe points (variables)Memory, # RPCs, # Failures, etc.Monitor collects time series of those values and computes functions over themDashboards prepare information graphicallyMostly used for diagnosis by humansJobMonitor
- 30. White-Box Monitoring:How itsdone @ GoogleJobMonitorDeclarative language for time series computationsCollects samples from the server by memory scrapingMerging of similar data from multiple servers running the same jobRich support for diagram rendering in the browser
- 31. White-Box Monitoring:Issues?Design formonitorability/testability?Its already ubiquitous throughout, since software engineers are themselves on-call…Distributed collection/network load?Not really an issue because it’s sample basedRelation to testing?Same as with black-box – should be a common framework.Automatic root cause analysis and self-repair?Current systems mostly build for human analysis and repair. Self-repair would be a big thing.JobMonitor
- 32. ChallengeSelf-repairJobMonitorCloud system arehomogenous and operate with redundancyMany VMs with exactly the same propertiesSelf-repair could identify and ‘drain’ faulty parts of the system, apply fallbacks, rollback software updates, etc.One major cause of cloud failures are outagesAnother major cause are software updates A semi-automated approach suggesting actions to a human would be already very usefulEver got paged at 2am in the morning?
- 33. ChallengeHybrid White-Box Monitoring/ RVFoundationsJobMonitorThe data collected in white-box monitoring represents continuous and often stochastic functions over time. The triggers for discrete actions (like alerts) are thresholds over integrated values of those functions.Sounds like hybrid systems/automaton. Has anybody looked at it like this in RV community?
- 34. Log AnalysisJobLogsCollect datafrom each server’s run containing information like operation flow, exceptions, etc.Store data over a window of time (say for last 24h)Access data from various sources programmatically to analyze issues (post-mortem, performance, etc.)Allows for correlation of system/service wide information
- 35. Log Analysis:How itsdone @ GoogleJobLogsVery fined grained logging on job side; huge amounts of data collectedLogs are stored in Bigtable (Google’s large-scale storage solution)Logs are analyzed using parallel (cloud) computing, e.g. with Sawzall, a declarative language based on map-reduceLogs are most often used for failure cause analysis/debugging
- 36. Log Analysis:Issues?JobLogsAmount ofdata and accessibility?Not really an issue because of highly performing distributed file systemsFormat of the data?Logs are structured data (at Google, protocol buffers)Encryption?A big issue: if it can’t be decrypted, not much may be diagnosable. If it can be decrypted, the access to this now decrypted data needs to be restricted.
- 37. ChallengePrivacy and EncryptionJobMonitorDatalogged (or otherwise analyzed) during monitoring may contain encrypted proprietary informationA problem may not be diagnosable without decryptionDecrypted clear text data (in particular if logged) needs to be highly protectedAutomatic obfuscation and/or anonymization would be highly desirableA protocol may need to be designed for this in the first place
- 38.
- 39.
- 40. ChallengeIntegration TestingJobJobJobStorageTwo ormore components are plugged together with a partially mocked environmentThese tests are usually very ‘flaky’ (unreliable) because:Difficulty to construct mocked component’s precise behavior (its more than a simple mock in a unit test)Difficulty to synthesize mocked component’s initial state (it may have a complex state)Potential solution: model-based testing
- 41. Model-Based Testing ina NutshellRequirementsFeedback AuthorModelFeedback Feedback GenerateTest SuiteIssueExpected Outputs (Test Oracle)Inputs(Test Sequences)VerdictControl Observe Feedback Implementation
- 42. Technical Document TestingProgram of Windows: A Success Story for MBT222 protocols/technical documents tested22,847 pages studied and converted into requirements36,875 testable requirements identified and converted into test assertions69% tested using MBT31% tested using traditional test automation66,962 person days (250+ years)Hyderabad: 250 test engineersBeijing: 100 test engineers38
- 43. Comparison MBT vsTraditionalIn % of total effort per requirement, normalizing individual vendor performance
- 44. Vendor 2 modeled85% of all test suites, performing relatively much better than Vendor 139Grieskamp et. al: Model-based quality assurance of protocol documentation: tools and methodology. Softw. Test., Verif. Reliab. 21(1): 55-71 (2011)
- 45. Exploiting the Cloudfor Development
- 46. Idle Resources Peak demandproblem: as with other utilities, the cloud must have capacity to deal with peak times: 7am, 7pm, etc.Huge amounts of idle computing resources available in the DCs outside of those peak timesLiterally hundreds of VMs may be available for a single engineer on a low-priority job baseGame changer for software development toolsUsing the Cloud for Dev @ GoogleDistributed/parallel buildEvery engineer can build all of Google’s code + third party open source code in a matter of minutes (sequential build would take days)Works by constructing the dependency graph than using map/reduce technologyDistributed/parallel testChanges on the code base are continuously tested against all dependent targets once submittedFailures can be tracked down very precisely to the given change which have introduced themCheck out http://google-engtools.blogspot.com/ for details
- 47. Consequences for Testing,Program Analysis, etc. Need to rethink the base assumptions of some of the existing approaches for testing and program analysis for massive coarse-grained parallelism:Early divide-and-conquer ideal, e.g. initial random seed than run-till-end and collect and compareTry different heuristics on the same problem; see which one winsTechniques like SMT and concolic execution can largely benefit from this
- 48. A formal methodguy’s dream…
Editor's Notes
- #3 Formal method guyMSR: AsmL, Spec#, Spec ExplorerWindows: Improving Microsoft Documentation (Open Documents) as resulting regulatory scrutiny of Microsoft. Using Model-Based Testing on a large scale in a project w/ over 300 person years of test effortSince April: Engineer @ Google. Currently working on G+ platform and tools. Just graduated from what they call a ‘Noogler’ (new Googler) ramping up on Google technology. To be clear: What I’m saying here is my personal viewpoint not necessarily inline with Google’s official opionion or direction. My intend is to not show you some research results or technologies as they come out of Google. Rather I’m trying to extract some challenges and opportunities for runtime verifcation research from my personal viewpoint on how things work @ Google. Other people @Google or in the industry might have totally different viewpoints.Also note that some of the technology used in the Google stack is confidential, others well-known or even open source. When it comes to the confidential parts, I will need to abstract and extract the underlying abstract concepts. So don’t take me literally when talking about the Google stack.
- #4 Will generally talk about what cloud computing is, how the market looks like. There is a lot of confusion about cloud computing, and I’ll try to set some scope. This maybe cold coffee for many but perhaps not for all.The largest part of this talk, will talk about monitoring (or RV or RA or however you want to call it) the cloud. This will present how a data center works, and how Google uses monitoring techniques to run it. It will pinpoint where there IMO no issues as the existing technologies work surprisingly well. It will pinpoint where I see gaps and challenges for research.I will shortly talk about testing the cloud. The biggest challenge here is integration testing, for which I suggest MBT. In this context I will also shortly talk about my experience with MBT when working for MS.I will dig into how the cloud can be actually exploited for software development. There are exciting opportunities here which wait to be applied.I will insert a small plug about a personal vision or dream around languages and specificationConclusion
- #6 Computation as a serviceIts like a utilityBased on shared resources
- #7 The picture you find on Wikipedia. It gives an idea of the components:Applications/SoftwarePlatformInfrastructure (Hardware)We dig into some of this in detail later on
- #8 Some properties of cloud computing as a user experiences it.
- #9 Let me talk a little bit about the cloud stack from the Market perspective. Most information I present here is extracted from an excellent article in the Economist (including this nice picture)
- #10 The three segments of the cloud stackSAAS‘PARSE’Eye-ASS
- #11 [Pronouncedeye-ass]Provides the hardware layers (data centers)For Google, one large one is for example in The Dalles Oregon. Size of two football fields, cooling towers four stories high, etc.A very important property is homogeneity of the hardware, often also achieved by using VMs. That makes it possible to migrate jobs in and between DCsThere are a number of big players. The actual size of the DCs (in number of machines) is highly confidential for each player. Guess why? …. A big issuein IAAS is the allocation problem, i.e. how many machines are required to provide certain services. The various players have there own ‘black magic’ to compute this.The revenue numbers are taking from the Economist article. The actual revenue is not disclosed directly by the players, so this is an estimate of the author of this article, and I have no idea about its accuracy… (In particular, this is not a number from Google!) The number looks actual relative small. This maybe related to that IAAS is not usually sold ‘as is’ but the actual end products – PAAS or SAAS – is sold.
- #12 Pronounced parseThis is basically the operating system plus frameworks and development tools. Not too many players in this space. For Google, it’s the app engine framework, which allows you to create and place applications in the Google cloud. For Microsoft, its Azure and Visual Studio.Estimated revenue of the whole market: again relative small. For companies like MS, platform is more a strategic investment which pays off indirectly.
- #13 Pronounced SARSNow this is how a user actually experiences the cloudThere are many players hereThe business is the largest
- #14 After this introduction, let us get a bit more technical and talk about monitoring of the cloud
- #15 The notions Monitoring, RV, RA, Testing, are often used to name similar or related things. What are the differences?Here is an attempt to make a definition.However, as in practice the boundaries are not so clear, and in particular at Google, monitoring is often used where other people would say RV, I will identify RV and monitoring.Testing is still a subject by itself though there is a lot of overlap.
- #16 Lets take a closer look how a DC actually works.If some sends a request on a domain like google.com, then the first thing is that a regional DNS will resolve this to particular DC closest to the locationIt reaches there some controller which will forward the request in kind of a hierarchy until a particular server (VM) is reached which handles the requestNote that high performing NFS is very important in a DC, so machines share a common storageMachines may be further organized in racks which may have certain replicated resources for shared storage
- #17 A server (or VM) usually runs a number of jobs (processes). Certain jobs which highly interact with each other may be arranged to run on the same serverIf it comes to monitoring, usually each (or a number) of jobs has associated a dedicated process which monitors the health of this job,The monitors collect data and can send alerts to user instances in the systen, alert managers.
- #18 A service (in contrast to a server) is about actually serving the initial user requestIt usually splits the task in sub-requests which are served by other jobs, often called backends. This is the major source of complexity in managing this kind of software.For certain activities, there maybe hundreds of jobs involved to get one request finally served
- #19 Now lets take a closer look at how monitoring works @Google. Black box: checks the health and basic functionalityWhite box: provides access to the internal state of a job collecting time series of dataLog analysis: processes after the fact logged data
- #22 Lets look at what are the issues with BB monitoring (if any). Where does it work and where not?
- #23 This is one problem. The worlds of test and of BB monitoring are largely disjoint.This maybe partly because of the original different engineering disciplines of operating engineers (called SREs) and software engineersIt would be nice to run some of the monitoring rules already at test time. And it would be nice to run of the test cases at monitoring time.It’s a matter of setting up a framework like Junit to decorate test cases for monitoring and provide other required meta dataNot really rocket science, though
- #24 Monitor what actually goes wrong when a request is serviced, following all its way to the topologyActually not important to catch failures; this works alreadyMaybe more important to analysis causes of failures and potentially prevent them before things crash