Muves3 Elastic Grid Java One2009 Final

2,638 views

Published on

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
2,638
On SlideShare
0
From Embeds
0
Number of Embeds
6
Actions
Shares
0
Downloads
31
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide
  • Title: How Cloud Computing and the NetBeans Rich Client Platform Enable the Army Research Laboratory’s Next-Generation Simulation System This presentation will provide an overview of how the Army Research Laboratory's (ARL’s) MUVES 3 project is using cloud computing and the NetBeans Rich Client Platform (RCP). The MUVES 3 architecture will be discussed along with the Java technologies that are used to develop it. The presentation will also show how the combination of a NetBeans RCP client application with a dynamic, service-oriented architecture (SOA) back-end can be used to construct sophisticated, high-performance environments. It will also discuss the MUVES 3 team’s experiences performing continuous integration using Hudson and testing the system on Amazon’s Elastic Compute Cloud using Elastic Grid. ARL analyzes combat system survivability and munition lethality against enemy systems. The goal of the MUVES 3 project is to develop an integrated environment to support this analysis mission. On the server side, the MUVES 3 system must be capable of executing concurrent simulation jobs to support a large analyst community. Although national security concerns prohibit running an actual MUVES 3 analysis on the cloud, the system architecture can be tested on it. This enables ARL to test MUVES 3 and evaluate system performance, scalability, and fault tolerance across numerous computers cost-effectively. The NetBeans RCP provides the framework upon which the MUVES 3 user interfaces are built. It is used both as a client platform to the cloud and as a client to the massively scalable simulation system.
  • Title, tweaked words, new top images
  • “ Animate” this to highlight each in turn?
  • “ Animate” this to highlight each in turn?
  • “ Animate” this to highlight each in turn?
  • “ Animate” this to highlight each in turn?
  • “ Animate” this to highlight each in turn?
  • Muves3 Elastic Grid Java One2009 Final

    1. 1. Cloud Computing and the NetBeans IDE Enable the Army Research Laboratory’s Next-Generation Simulation System. Ron Bowers Army Research Laboratory Dennis Reedy Elastic Grid LLC. 3 June 2009
    2. 2. Agenda <ul><li>Where we work and what we do </li></ul><ul><li>Overview of the architecture </li></ul><ul><li>Why and how we are using Cloud Computing </li></ul><ul><li>Next steps and follow up </li></ul>
    3. 3. The Army Research Laboratory <ul><li>The Army Research Laboratory (ARL) is the Army's corporate basic and applied research laboratory. Our mission is to provide innovative science, technology, and analysis to enable full-spectrum operations. </li></ul><ul><li>We represent the Survivability/Lethality Analysis Directorate (SLAD) of ARL. </li></ul>
    4. 4. We have some experience with computers...
    5. 5. <ul><li>Ensure that US personnel and equipment… </li></ul>SLAD Mission Nuclear, Biological and Chemical Attack Ballistic Threats Electronic Warfare Information Warfare … survive and function effectively in hostile circumstances
    6. 6. SLAD performs both experimentation and modeling
    7. 7. SLAD Ballistic Vulnerability/Lethality (V/L) Modeling <ul><li>SLAD’s primary tool for performing ballistic V/L analysis is MUVES. </li></ul><ul><li>MUVES development began in 1984. </li></ul><ul><li>The current version (2.16) is a single-threaded C application. </li></ul><ul><li>We are currently developing MUVES 3, which is an all-new replacement system. </li></ul>
    8. 8. Program Focus <ul><li>Provide the next generation of simulation system for the V/L analyst community </li></ul><ul><ul><li>Mostly Java. </li></ul></ul><ul><ul><li>Dynamic distributed and service-oriented. </li></ul></ul><ul><ul><li>Will support over 100 concurrent users. </li></ul></ul><ul><ul><li>Incorporates a computational grid, parallelized system that distributes tasks and computes results that are graphically displayed. </li></ul></ul><ul><ul><li>Will operate in both “batch” and interactive modes. </li></ul></ul>
    9. 9. Interesting Challenges <ul><li>We have few servers, but many powerful workstations. </li></ul><ul><ul><li>Architecture must exploit analyst community machines </li></ul></ul><ul><ul><li>Share CPU, memory and disk </li></ul></ul><ul><ul><li>Heterogeneous deployment environment </li></ul></ul><ul><li>Required application assets vary </li></ul><ul><ul><li>Need real-time provisioning of application assets </li></ul></ul><ul><ul><li>Must be able to route functionality to machines that are best capable of executing tasks/functions </li></ul></ul><ul><ul><li>Must be able to scale on demand based on real time need and use of the system </li></ul></ul><ul><li>Legacy of performance issues and nightmares </li></ul>
    10. 10. Solution Approach <ul><li>Choose technology that embraces dynamic distributed capabilities </li></ul><ul><li>Craft a loosely coupled service oriented architecture that segments the system into functional roles </li></ul><ul><li>Choose persistence technologies and approaches that allow for low latency and high concurrency </li></ul><ul><li>Represent data as it moves through the system </li></ul><ul><ul><li>In-flight (hot in-memory), Swap, Long Term, Archived </li></ul></ul><ul><li>Keep Disk I/O out of the main stream processing </li></ul>
    11. 11. What’s Underneath Application Infrastructure Quality of Service Monitoring and Management Domain-specific Services and Algorithms Dynamic Container Persistence Management
    12. 12. What’s Underneath Application Infrastructure Quality of Service Monitoring and Management Domain-specific Services and Algorithms Dynamic Container Persistence Management JavaSpace Apache Active MQ Apache Derby
    13. 13. What’s Underneath Application Infrastructure Quality of Service Monitoring and Management Domain-specific Services and Algorithms Dynamic Container Persistence Management Rio
    14. 14. What’s Underneath Application Infrastructure Quality of Service Monitoring and Management Domain-specific Services and Algorithms Dynamic Container Persistence Management Gomez
    15. 15. Gomez <ul><li>Was established as a prototype for MUVES 3 architectural enhancements. </li></ul><ul><li>Now forms the basis of the MUVES 3 service-oriented architecture (SOA). </li></ul><ul><li>Includes all of the non-sensitive services used by MUVES 3. </li></ul><ul><li>Is an open source project (LGPLv3). </li></ul>
    16. 16. What’s Underneath Application Infrastructure Quality of Service Monitoring and Management Domain-specific Services and Algorithms Dynamic Container Persistence Management
    17. 17. MUVES 3 System Organization <ul><li>Attachment point for clients. </li></ul><ul><li>Monitors system load. </li></ul><ul><li>Controls job submission. </li></ul><ul><li>Executes analysis </li></ul><ul><li>jobs. </li></ul><ul><li>Stores analysis </li></ul><ul><li>results. </li></ul>Gateway Sim Persistence Client MUVES 3 UI
    18. 18. MUVES 3 UI <ul><li>The MUVES 3 UI is built on the NetBeans Platform. </li></ul><ul><li>Used for: </li></ul><ul><ul><li>Input preparation </li></ul></ul><ul><ul><li>Team collaboration </li></ul></ul><ul><ul><li>Job submittal and monitoring. </li></ul></ul><ul><ul><li>Result visualization </li></ul></ul><ul><li>Components that interact with back-end services are developed in Gomez and used in the MUVES 3 UI. </li></ul>
    19. 19. Our NetBeans Experience <ul><li>Favorite things </li></ul><ul><ul><li>Swing-based </li></ul></ul><ul><ul><li>Fast form development </li></ul></ul><ul><ul><li>Easy deployment via JNLP </li></ul></ul><ul><ul><li>Extremely well supported by the community </li></ul></ul><ul><li>Biggest issue </li></ul><ul><ul><li>Integrating libraries that are updated frequently </li></ul></ul>
    20. 20. MUVES 3 Execution Client Gateway Persistence Sim Pool Busy Sim Pool Sim Pool
    21. 21. MUVES 3 Execution Client Gateway Select Sim Pool Persistence Sim Pool Busy Sim Pool Sim Pool Submit job
    22. 22. MUVES 3 Execution Client Gateway Sim Pool Persistence Worker Worker Worker Worker Job Monitor Task Space
    23. 23. MUVES 3 Execution Client Gateway Sim Pool Persistence Ray Tracer Personnel Vulnerability Specialized Physics Vehicle Performance Worker Worker Worker Worker Job Monitor Task Space Deploy additional services
    24. 24. MUVES 3 Execution Client Gateway Sim Pool Persistence Ray Tracer Personnel Vulnerability Specialized Physics Vehicle Performance Worker Worker Worker Worker Job Monitor Task Space Submit job Store results Visualize Results
    25. 25. Dynamic Clustering Client Proxy Service Interface Service <ul><li>Available strategies: </li></ul><ul><li>Fail-Over – Uses one service unless that service goes away. </li></ul><ul><li>Round-Robin – Iterates over all discovered services. </li></ul><ul><li>Utilization – Like round-robin, but ignores services that are low on system resources. </li></ul>Service Service Rio Association declaration Service discovery Service injection Service Selection Strategy Proxy injection
    26. 26. The Persistence Meta-service <ul><li>Stores the results of analyses. </li></ul><ul><li>Consists of four layers: </li></ul><ul><ul><li>In-memory cache (distributed JavaSpace), </li></ul></ul><ul><ul><li>Swap, (Apache Active MQ) </li></ul></ul><ul><ul><li>Long-term storage (Apache Derby), and </li></ul></ul><ul><ul><li>Archive (Hibernate+Oracle). </li></ul></ul><ul><li>Layers are implemented using dynamic clustering. </li></ul><ul><li>Supports data life-cycle management </li></ul><ul><ul><li>Data storage is leased, and leases can expire. </li></ul></ul><ul><ul><li>When the cache fills, results are moved to swap. </li></ul></ul><ul><li>Must work for the next 20 years. </li></ul>
    27. 27. Cloud Computing <ul><li>Public Cloud: </li></ul><ul><ul><li>Obvious national security issues </li></ul></ul><ul><li>Private Cloud </li></ul><ul><ul><li>Conceptually already there </li></ul></ul><ul><ul><ul><li>Realtime provisioning of applications </li></ul></ul></ul><ul><ul><ul><li>Dynamic allocation </li></ul></ul></ul><ul><ul><ul><li>SLA based approach </li></ul></ul></ul>
    28. 28. Cloud Computing:Applicable for Testing? <ul><li>Goals </li></ul><ul><ul><li>Demonstrate the performance and scalability of MUVES 3 over dozens (or hundreds :-) ) of computers. </li></ul></ul><ul><ul><li>Execute multiple integration tests concurrently. </li></ul></ul><ul><li>Issues </li></ul><ul><ul><li>Small number of computers available locally for testing. </li></ul></ul><ul><ul><li>Difficult coordination issues due to Army security policies. </li></ul></ul><ul><li>Approach </li></ul><ul><ul><li>Test the MUVES 3 architecture (Gomez) on Amazon Elastic Compute Cloud (EC2) </li></ul></ul>
    29. 29. Cloud Adoption Challenges <ul><li>Getting approval :) </li></ul><ul><li>Administrative burden </li></ul><ul><ul><li>We don’t want to build AMIs or go through the time to provision an entire stack every night. </li></ul></ul><ul><li>Minimize changes </li></ul><ul><ul><li>Avoid developing special code and testing framework for cloud deployment/orchestration. </li></ul></ul><ul><ul><li>Ideally, transparently switch from LAN-based deployment to the cloud. </li></ul></ul><ul><li>We must preserve the dynamic distributed semantics architected in the system: </li></ul><ul><ul><li>Service selection strategies </li></ul></ul><ul><ul><li>Dynamic discovery semantics </li></ul></ul><ul><li>Run multiple concurrent test cases and roll up test results. </li></ul>
    30. 30. Cloud Adoption Approach <ul><li>Use Elastic Grid (EG) </li></ul><ul><ul><li>Eases development and deployment of Java applications into the Cloud </li></ul></ul><ul><ul><li>Provides automated management, fault detection, and scalability for the application </li></ul></ul><ul><ul><li>Allows focus on development, not cloud infrastructure </li></ul></ul>
    31. 31. Elastic Grid Overview <ul><li>Cloud Management Fabric </li></ul><ul><ul><li>Dynamically instantiate, monitor & manage application components </li></ul></ul><ul><ul><li>SLA policy driven with strategies like service scalability, relocation, fault detection & recovery, etc. </li></ul></ul><ul><li>Cloud Virtualization Layer </li></ul><ul><ul><li>Abstracts specific Cloud Computing provider technology </li></ul></ul><ul><ul><li>Allows portability across specific implementations </li></ul></ul>You can deploy on Amazon EC2, Private LAN based Cloud; Eucalyptus, Sun Cloud and others soon
    32. 32. Cloud Activation & Deployment Groovy Client Application Monitors Application Agents 2 Create Clusters Download and deploy application resources Deploy 3 Upload JUnit test results 4 Build, Create Release, Upload Archive S3 1 Download Test results and post process 5
    33. 33. Cloud Activation & Deployment Test Cluster Test Cluster Test Cluster Groovy Client S3
    34. 34. Extending Continuous Integration <ul><li>Automated build and test, both unit and integration tests </li></ul><ul><li>Extend this to include Continuous Deployment </li></ul><ul><ul><li>If CI passes, use Test Cloud Bursting to deploy, verify and validate system </li></ul></ul>SVN Repo. Deploy system and run tests Retrieve and process test results Test Runner
    35. 35. Next Steps <ul><li>Lots to think about </li></ul>
    36. 36. Summary <ul><li>Using the public cloud for scalability testing and verification a good choice </li></ul><ul><li>Without Elastic Grid we would have a much more difficult experience (may not have done it) </li></ul><ul><li>Looking to expand metrics gathering & collection </li></ul><ul><li>Looking to incorporate NetBeans as a client for cloud visualization </li></ul><ul><li>We are looking to make this part of our permanent development environment </li></ul><ul><li>How to spread the gospel of cloud computing for DoD </li></ul>
    37. 37. Dennis Reedy Elastic Grid LLC [email_address] Ron Bowers Army Research Laboratory [email_address]

    ×