Azure Monitor & Application Insight to monitor Infrastructure & Application
Nectar cloud workshop ndj 20110331.2
1. Thoughts from the east, on research in the cloud NeCTARResearch Cloud Workshop30 – 31 March 2011, Melbourne, Australia Nick Jones n.jones@auckland.ac.nz www.eresearch.auckland.ac.nz
13. NZs eResearch Infrastructures HPC & Distributed Computing BeSTGRID, since 2006 $2.5M + $800k Largest national grid users in Australasia Built capability, established a model for sharing resources NZ eScience Infrastructure (NeSI) ~$50M ($27M Crown) 4 HPC centres across NZ Signing agreements now, running for 4 years > review > ongoing… NZ Genomics Ltd (NZGL) National genomics service 4 major partners Funded in 2008 Almost operational..
42. Land of the long white cloud.. Within NZGL and NeSI, we have reasonable scale investments into virtualised infrastructure Our aim is to build at least a federated cloud platform, if not a single research cloud (perhaps with mulitple availability zones)
44. Who are our users? Researchers HPC users System Administrators Application Developers What are the funders expectations? Facebook for scientists? VM and development platforms Scaled out server infrastructure & development platform efficiencies and coordination National collaborative infrastructure Needs and solutions depend on which layer of the cloud you’re creating: SaaS, PaaS, IaaS
45. *aaS Examples Galaxy (Genetic Marker Design) analysis modules workflows Biocommons multisite sheep.biocommons.org.nz (Genomics) analysis modules pipelines Scale out analyses? Self service Customised / templated services Reuse of methods?
48. Joining the Grid to the Cloud Issues, early 2010: Maturity of solution Overhead of VM provisioning Cloud Workshop 18/02/2011 https://wiki.heprc.uvic.ca/twiki/bin/view/Main/CloudWorkshop Examines queue, launches VMs on Nimbus, EC2, Eucalyptus Let users define the computational image Provide a library for them to share, discover, launch
50. Users Issues & Needs What are their issues? Accessibility Usability Domain specificity Funding sources (opex, capex, none) Sustainability What are their needs? Self service / on demand Applications Registry Ease of discovery Consultancy Migration, adaptation Mature Service Delivery
51. Come… meet the cloud….. Manapouri underground power station
53. CONSOLE OPERATOR AT SAIGON SATELLITE TERMINAL. Soldier monitors satellite traffic and selects satellite to be used.
54. eResearch Tools development In the past, staff have been systems operators and middleware developers few business analysts and user centered designers Who is driving product and service development? Who discovers needs, translates into eResearch Tools?
55. Where are the users now? Under their desks! In institutional facilities In externally supported communities Migrating users to the Cloud And, we want them to join us!
56. Migrating eResearch Tools Monolithic systems aren’t compatible with HPC, Grid, nor the Cloud Applications require scaling out, whether databases, web based analysis engines, desktop applications
57. Going the last mile? Who is supporting the researcher, to understand their needs, define their requirements, and implement their solutions? Are there services and applications either already in place, or under development, to meet needs, and fill capacity created? Who will cross the boundaries between these groups?
58. Who are our users? Who are the users? Researchers – technically savvy ones.. HPC users.. maybe System Administrators Application Developers “All researchers that need to use eresearch tools already know how to and are doing so..” “If you can’t write the code yourself, how can you trust the results? Surely you must be able to write the code!?”
59. What do I talk about, when I talk about Cloud? Instead of the machinery: VMs, hypervisors, schedulers, block storage, virtual networks, hosts, networks, firewalls How about capabilities: Scalability: Methods, Communities Self Service Sustainability(through preservation) Where do we do this well? Data? Preservation Curation HPC? Algorithms Methods Hardware optimisation techniques When communicating to end users, common vocabularies in other communities aren’t about the machinery, they’re higher level…
60. What do I talk about, .. Scalability What is the equivalent of algorithms and optimal approaches in the Cloud era? The knowledge, skills, and methods to optimally use the cloud? How to be elastic (scale up/down, on demand) How to manage failure Multi-tenancy Building communities, leveraging network effects
61. What do I talk about, .. Preservation What are the research processes and artefacts we are interacting with? Methodologies.. enshrined as VMs? e.g. research codes, which take cuts and bruises to re-instantiate = knowledge artefacts that have value and are scarce What about workflows? Shouldn’t we wait until we have enshrined these methods into sophisticated workflows, and ontologies? .. So we should throw away any method until it reaches a predefined threshold of maturity? Because before then, it has no value? Oh, so all methods are important? that’s up to the researcher to say.. Method construction and archive, self service. Ahh, so we’re close to being able to archive and preserve rich research methods? for those that enshrine their methods in software based systems… YES! Reuseable workflows and services for large collaborative communities? … like Virtual Organistions, they only suit certain levels of scale and maturity Individual research codes, workflows, pipelines = research notebooks for those who efficiently enshrine their thoughts and ideas in code These are people we should support! And, their experiments things we should preserve..
62. What did we learn from the grid? So.. …. what do we need to get right, to be successful?
63. Get the team right Composition & Coordination of distributed team essential Fully committed FTEs Cross site responsibilities, including shared systems administration Strong leadership and programmemanagement
64. Strong guidance Which resources are getting used, why? Prescriptive framework necessary to ensure participation … need a well defined service model
65. Should we build new institutions? Why do we want to build institutions? Scale efficiencies Sustainability .. and reliable targeted accountable services We often take project funding, and aspire to build new institutions Creating institutions is and will continue to be expensive and difficult.. Our institutions are long lived, and can absorb most technologies, given care and enough time to mature
66. Really think about Coordination Why do we need Government money? Scale? We’re partly seeing a coordination failure (coordination of capital and resources) It is the incentive to bring out institutions into alignment.. Funding: work with funders to align opex and capex. Crown and coinvestors fund both in NZ, clarifying intent of investments Otherwise, whoever funds the operating, eats the cake Ensure KPIs incentivize sustainability Incentives and obligations … clarify expectations to coinvestors, sector “Single Front Door” Coordinate with broadest scale IT community that has responsibility and capability Research labs can be good, though often will be introspective / focus on local solutions / research outputs Drive innovations into core IT Services groups; strong coordination from top level sponsors required Collaborate nationally and internationally
67. Usability really counts !! Make it easy! Authentication … Authentication … Authentication … Consistent User Interfaces? Don’t get in my way Don’t break the abstraction Don’t give me middleware!!
68. Build communities Is the aim to build aggregations of communities? Support national scale communities? … but these may not exist will need to build communities (developers, end users, administrators, leaders) Who will mediate within and between communities, seek out commonalities, and take ownership of developments?
69. Support the long tail There’s an endless list of applications and services, that individual researchers need and want We often focus on the large communities, common platforms, and scalability To make a Research Cloud useful for the majority of the community, we need to design for great diversity We can do this incrementally We’re looking to learn, fast, what works for researchers, and what doesn’t >>> we need agility, and very strong engagement with users communities
70. Thoughts from the east, on research in the cloud Thanks NeCTAR Research Cloud Workshop30 – 31 March 2011, Melbourne, Australia Nick Jones n.jones@auckland.ac.nz www.eresearch.auckland.ac.nz
71. Image Credits www.nzhistory.net.nz, the website of NZHistory.net.nz, New Zealand history online, at ManatūTaonga. Licensed by ManatūTaonga for re-use under the Creative Commons Attribution-Non-Commercial 3.0 New Zealand Licence. www.history.army.mil, Communications Electronics 1962-1970, Department of the Army www.travelpod.com, Travel Blog, 7.08.2010, http://www.travelpod.com/members/wbeardsl
72. Goals Needs of users, as identified at this workshop Policy implications Applications & Services that are in common national usage Enable provision of excellent cloud services to researchers Nodes operating within a prescribed framework Framework Consistent user interface to the range of applications and services running at the distributed nodes Applications and Services Selected by Panel Data analysis, visualisation, collaboration, security, application and service platforms, portals / interfaces to HPC and commercial cloud providers Infrastructure as a Service Hosting Platform as a Service offerings for research communities Distributed Nodes Selected by panel Prescriptive operating model Lead Node will create & operate framework; monitor service delivery Access, interop, application migration, security, licensing, accounting, implementation practicalities, monitoring and maintenance Common layer that each node can interoperate with
![Genomics, Genetics, Bioinformatics, Molecular Modelling: NZ Genomics Ltd Maurice Wilkins Centre Alan Wilson Centre Virtual Institute of Statistical Genetics Wind Energy, Geothermal & Minerals Exploration: ,[object Object]](data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)