ICEOTOPE & OCF: Performance for Manufacturing IceotopePR
ICEOTOPE, OCF & The Advanced Manufacturing Research Centre (AMRC) define the performance required for a manufacturing environment and potential challenges to overcome in order to enable a faster time-to-market.
This presentation was given at the London Nutanix user group (NUG) on Oct 26 by Ray Hassan. If you would like to join a NUG, you can find more information here http://bit.ly/NTNXUG - Hope to see you at a community meeting!
Hadoop is an open-source software framework for storing data and running applications on clusters of commodity hardware. It provides massive storage for any kind of data, enormous processing power and the ability to handle virtually limitless concurrent tasks or jobs.
SUSE Enterprise Storage - a Gentle IntroductionGábor Nyers
SUSE Enterprise Storage is a scalable and resilient software-based storage solution. It lets you build cost-efficient and highly scalable data storage using commodity, off-the-shelf servers and disk drives.
Red Hat Ceph Storage: Past, Present and FutureRed_Hat_Storage
Ceph is a massively scalable, open source, software-defined storage system that runs on commodity hardware. Get an update about the latest version of Red Hat Ceph Storage, including information about the newest features and use cases, with a particular focus on cloud storage and OpenStack. We’ll also explore the themes and directions for the roadmap for the next 12 months.
ICEOTOPE & OCF: Performance for Manufacturing IceotopePR
ICEOTOPE, OCF & The Advanced Manufacturing Research Centre (AMRC) define the performance required for a manufacturing environment and potential challenges to overcome in order to enable a faster time-to-market.
This presentation was given at the London Nutanix user group (NUG) on Oct 26 by Ray Hassan. If you would like to join a NUG, you can find more information here http://bit.ly/NTNXUG - Hope to see you at a community meeting!
Hadoop is an open-source software framework for storing data and running applications on clusters of commodity hardware. It provides massive storage for any kind of data, enormous processing power and the ability to handle virtually limitless concurrent tasks or jobs.
SUSE Enterprise Storage - a Gentle IntroductionGábor Nyers
SUSE Enterprise Storage is a scalable and resilient software-based storage solution. It lets you build cost-efficient and highly scalable data storage using commodity, off-the-shelf servers and disk drives.
Red Hat Ceph Storage: Past, Present and FutureRed_Hat_Storage
Ceph is a massively scalable, open source, software-defined storage system that runs on commodity hardware. Get an update about the latest version of Red Hat Ceph Storage, including information about the newest features and use cases, with a particular focus on cloud storage and OpenStack. We’ll also explore the themes and directions for the roadmap for the next 12 months.
(ATS4-PLAT06) Considerations for sizing and deploymentBIOVIA
The Accelrys Enterprise Platform (AEP) installation guide provides a basic set of system requirements for your server. While these values will indeed get you up and running, they may not be adequate for your usage scenario. Correctly sizing your server is critical for your Enterprise’s successful deployment of AEP. There is no one-size-fits-all answer for how big your server should be, so we will cover multiple scenarios and considerations to keep in mind. Following these guidelines, your Enterprise deployment will more successful and your users will be happier. Specific scenarios covered may include Application (BioReg, ChemReg) deployment, Next Generation Sequence usage, and Modeling and Simulation usage
24 Hours of PASS, Summit Preview Session: Virtual SQL Server CPUsDavid Klee
One of the largest points of contention with virtual SQL Servers and the VM administrators is how to configure the CPUs. Experience says more CPUs are better for performance. VM admins say less is better. Third-party vendors say you need all of them (and it doesn’t matter how many your hosts have either). Can over-provisioning virtual machine CPUs speed things up, or does it slow things down? What is the right methodology to determine the correct number of virtual CPUs? How does this configuration align with the physical servers? From sampling and analyzing performance data, to “right-sizing’ your SQL Server virtual machine CPU count, to properly aligning the VM with the physical server NUMA topology, you will gain the understanding of how to properly manage and validate your virtual SQL Server vCPU configuration in this insightful session. Valuable tips and tricks will be shared that you can take back to your virtual SQL Servers and immediately apply to your own environments.
A simple setup to build a private or public cloud.
A cloud at the IaaS layer is simply a cluster of hypervisors with some added storage infrastructure and software to orchestrate everything. In this presentation we show some straightfoward DELL hardware that could be purchased to build a single rack as the basic for a private or public cloud. It totals $100k and coupled with open source software: cloudstack, ceph, glusterfs, nfs etc is the basis for your cloud.
You will get a AWS compatible cloud in no-time and with limited acquisition cost.
OpenStack and Ceph case study at the University of AlabamaKamesh Pemmaraju
The University of Alabama at Birmingham gives scientists and researchers a massive, on-demand, virtual storage cloud using OpenStack and Ceph for less than $0.41 per gigabyte. This is a session at the OpenStack summit given by Kamesh Pemmaraju at Dell and John Paul at University of Alabama. This will detail how the university IT staff deployed a private storage cloud infrastructure using the Dell OpenStack cloud solution with Dell servers, storage, networking and OpenStack, and Inktank Ceph. After assessing a number of traditional storage scenarios, the University partnered with Dell and Inktank to architect a centralized cloud storage platform that was capable of scaling seamlessly and rapidly, was cost-effective, and that could leverage a single hardware infrastructure for the OpenStack compute and storage environment.
Azure en Nutanix: your journey to the hybrid cloudICT-Partners
Op zoek naar oplossingen voor een flexibel, schaalbaar, kostenefficiënt en toekomstvast datacenter? Ontdek dan nu de kracht van Microsoft Azure & Nutanix: twee moderne platformen waarmee u de voordelen van uw on-premise infrastructuur kunt combineren met de voordelen van de public cloud.
Presentatie van 30 april 2015
Protect the Hype: Backup Best Practices for Converged & Hyperconverged Infras...marketingunitrends
Infrastructure is getting smaller through convergence. Cisco UCS exploded in the market by reducing infrastructure components and footprint. Hyperconverged vendors like Nutanix are taking that concept even further – talking about “invisible infrastructure.” The shifts in delivery of storage, compute, networking and virtualization create huge opportunities to improve delivery and TCO, but they can also make you want to tear your hair out trying to protect it all.
Introducing Affordable HPC or HPC for the Masses - IBM NeXtScale System Cliff Kinard
Introducing IBM NeXtScale System -
The new IBM NeXtScale System™ is designed for flexibility, simplicity and scalability to lower operating costs for compute and data-intensive workloads such as technical computing, big data, and cloud. The system simplifies adoption through an open design that seamlessly fits into your current infrastructure. The NeXtScale System delivers the density, agility, and scale you need for today’s most demanding workloads.in a flexible cost optimized platform.
-Architected for now and the future
-Better density and flexibility
-Compatible with standard racks
-Optimized for top of rack switching
-Software Defined Networking (SDN) ready
-Intel® Xeon® Processor E5-2600 v2 product family
-Powerful roadmap
-Business partner friendly
Watch our North America webcast replay here:
http://event.on24.com/r.htm?e=670225&s=1&k=FC5CD17AB42385B40BCED29B8B61E2E8&partnerref=IBM09
Network support for resource disaggregation in next-generation datacentersSangjin Han
Presented at the 12th ACM Workshop on Hot Topics in Networks (HotNets XII)
Datacenters have traditionally been a collection of individual servers, each of which aggregates a fixed amount of compute, memory, storage, and network resources as an independent physical entity. Extrapolating from recent trends, we envisage that future datacenters will be architected in a drastically different manner: all computational resources within a server will be disaggregated into standalone blades, and the datacenter network will directly interconnect them. This is what we call "Disaggregated Datacenters".
This presentation briefly sketches why and how this transition will happen. In particular, we focus on the role of network fabric in such disaggregated datacenters, as it will face a set of challenges brought by the new datacenter architecture.
The paper can be found here: http://www.eecs.berkeley.edu/~sangjin/static/pub/hotnets2013_ddc.pdf
An overview of the development of the Apache Hadoop software stack, including some of the barriers to participation -and how and why to overcome them. It closes with some open discussion points/ideas of how the existing process can be improved.
(ATS4-PLAT06) Considerations for sizing and deploymentBIOVIA
The Accelrys Enterprise Platform (AEP) installation guide provides a basic set of system requirements for your server. While these values will indeed get you up and running, they may not be adequate for your usage scenario. Correctly sizing your server is critical for your Enterprise’s successful deployment of AEP. There is no one-size-fits-all answer for how big your server should be, so we will cover multiple scenarios and considerations to keep in mind. Following these guidelines, your Enterprise deployment will more successful and your users will be happier. Specific scenarios covered may include Application (BioReg, ChemReg) deployment, Next Generation Sequence usage, and Modeling and Simulation usage
24 Hours of PASS, Summit Preview Session: Virtual SQL Server CPUsDavid Klee
One of the largest points of contention with virtual SQL Servers and the VM administrators is how to configure the CPUs. Experience says more CPUs are better for performance. VM admins say less is better. Third-party vendors say you need all of them (and it doesn’t matter how many your hosts have either). Can over-provisioning virtual machine CPUs speed things up, or does it slow things down? What is the right methodology to determine the correct number of virtual CPUs? How does this configuration align with the physical servers? From sampling and analyzing performance data, to “right-sizing’ your SQL Server virtual machine CPU count, to properly aligning the VM with the physical server NUMA topology, you will gain the understanding of how to properly manage and validate your virtual SQL Server vCPU configuration in this insightful session. Valuable tips and tricks will be shared that you can take back to your virtual SQL Servers and immediately apply to your own environments.
A simple setup to build a private or public cloud.
A cloud at the IaaS layer is simply a cluster of hypervisors with some added storage infrastructure and software to orchestrate everything. In this presentation we show some straightfoward DELL hardware that could be purchased to build a single rack as the basic for a private or public cloud. It totals $100k and coupled with open source software: cloudstack, ceph, glusterfs, nfs etc is the basis for your cloud.
You will get a AWS compatible cloud in no-time and with limited acquisition cost.
OpenStack and Ceph case study at the University of AlabamaKamesh Pemmaraju
The University of Alabama at Birmingham gives scientists and researchers a massive, on-demand, virtual storage cloud using OpenStack and Ceph for less than $0.41 per gigabyte. This is a session at the OpenStack summit given by Kamesh Pemmaraju at Dell and John Paul at University of Alabama. This will detail how the university IT staff deployed a private storage cloud infrastructure using the Dell OpenStack cloud solution with Dell servers, storage, networking and OpenStack, and Inktank Ceph. After assessing a number of traditional storage scenarios, the University partnered with Dell and Inktank to architect a centralized cloud storage platform that was capable of scaling seamlessly and rapidly, was cost-effective, and that could leverage a single hardware infrastructure for the OpenStack compute and storage environment.
Azure en Nutanix: your journey to the hybrid cloudICT-Partners
Op zoek naar oplossingen voor een flexibel, schaalbaar, kostenefficiënt en toekomstvast datacenter? Ontdek dan nu de kracht van Microsoft Azure & Nutanix: twee moderne platformen waarmee u de voordelen van uw on-premise infrastructuur kunt combineren met de voordelen van de public cloud.
Presentatie van 30 april 2015
Protect the Hype: Backup Best Practices for Converged & Hyperconverged Infras...marketingunitrends
Infrastructure is getting smaller through convergence. Cisco UCS exploded in the market by reducing infrastructure components and footprint. Hyperconverged vendors like Nutanix are taking that concept even further – talking about “invisible infrastructure.” The shifts in delivery of storage, compute, networking and virtualization create huge opportunities to improve delivery and TCO, but they can also make you want to tear your hair out trying to protect it all.
Introducing Affordable HPC or HPC for the Masses - IBM NeXtScale System Cliff Kinard
Introducing IBM NeXtScale System -
The new IBM NeXtScale System™ is designed for flexibility, simplicity and scalability to lower operating costs for compute and data-intensive workloads such as technical computing, big data, and cloud. The system simplifies adoption through an open design that seamlessly fits into your current infrastructure. The NeXtScale System delivers the density, agility, and scale you need for today’s most demanding workloads.in a flexible cost optimized platform.
-Architected for now and the future
-Better density and flexibility
-Compatible with standard racks
-Optimized for top of rack switching
-Software Defined Networking (SDN) ready
-Intel® Xeon® Processor E5-2600 v2 product family
-Powerful roadmap
-Business partner friendly
Watch our North America webcast replay here:
http://event.on24.com/r.htm?e=670225&s=1&k=FC5CD17AB42385B40BCED29B8B61E2E8&partnerref=IBM09
Network support for resource disaggregation in next-generation datacentersSangjin Han
Presented at the 12th ACM Workshop on Hot Topics in Networks (HotNets XII)
Datacenters have traditionally been a collection of individual servers, each of which aggregates a fixed amount of compute, memory, storage, and network resources as an independent physical entity. Extrapolating from recent trends, we envisage that future datacenters will be architected in a drastically different manner: all computational resources within a server will be disaggregated into standalone blades, and the datacenter network will directly interconnect them. This is what we call "Disaggregated Datacenters".
This presentation briefly sketches why and how this transition will happen. In particular, we focus on the role of network fabric in such disaggregated datacenters, as it will face a set of challenges brought by the new datacenter architecture.
The paper can be found here: http://www.eecs.berkeley.edu/~sangjin/static/pub/hotnets2013_ddc.pdf
An overview of the development of the Apache Hadoop software stack, including some of the barriers to participation -and how and why to overcome them. It closes with some open discussion points/ideas of how the existing process can be improved.
A lightning talk on the benefits and issues with location tracking. Key point: we are giving invaluable information away for very little in return. Is this good or bad? Discuss.
Overview of the above and beyond MapReduce, for the HPC/science community. Key point: move up the stack, reuse what is there. But: some of these people are capable of writing their own YARN apps, so they should be encouraged to do so if they see a need.
Presentation on 2013-06-27, Workshop on the future of Big Data management, discussing hadoop for a science audience that are either HPC/grid users or people suddenly discovering that their data is accruing towards PB.
The other talks were on GPFS, LustreFS and Ceph, so rather than just do beauty-contest slides, I decided to raise the question of "what is a filesystem?", whether the constraints imposed by the Unix metaphor and API are becoming limits on scale and parallelism (both technically and, for GPFS and Lustre Enterprise in cost).
Then: HDFS as the foundation for the Hadoop stack.
All the other FS talks did emphasise their Hadoop integration, with the Intel talk doing the most to assert performance improvements of LustreFS over HDFSv1 in dfsIO and Terasort (no gridmix?), which showed something important: Hadoop is the application that add DFS developers have to have a story for
Cloud deployments of Apache Hadoop are becoming more commonplace. Yet Hadoop and it's applications don't integrate that well —something which starts right down at the file IO operations. This talk looks at how to make use of cloud object stores in Hadoop applications, including Hive and Spark. It will go from the foundational "what's an object store?" to the practical "what should I avoid" and the timely "what's new in Hadoop?" — the latter covering the improved S3 support in Hadoop 2.8+. I'll explore the details of benchmarking and improving object store IO in Hive and Spark, showing what developers can do in order to gain performance improvements in their own code —and equally, what they must avoid. Finally, I'll look at ongoing work, especially "S3Guard" and what its fast and consistent file metadata operations promise.
Explores the notion of "Hadoop as a Data Refinery" within an organisation, be it one with an existing Business Intelligence system or none - looks at 'agile data' as a a benefit of using Hadoop as the store for historical, unstructured and very-large-scale datasets.
The final slides look at the challenge of an organisation becoming "data driven"
Best Practices for Deploying Enterprise Applications on UNIXNoel McKeown
Gain some insight to a UNIX based operating system and the types of tasks a build team or vendors perform. How to prepare a UNIX server for typical enterprise deployments.
The best practices applied to a UNIX platform in a telco environment. Some hints, tips, troubleshooting and practical knowledge.
VMworld 2013: Virtualizing Databases: Doing IT Right VMworld
VMworld 2013
Michael Corey, Ntirety, Inc
Jeff Szastak, VMware
Learn more about VMworld and register at http://www.vmworld.com/index.jspa?src=socmed-vmworld-slideshare
There is a growing trend today of enterprises leveraging both Amazon Web Services (AWS) and on-premise OpenStack-based private clouds. However, the default networking option in OpenStack remains broken and the plethora of confusing plug-ins makes networking in OpenStack mysterious and difficult to manage.
Enter MidoNet, the open source network virtualization solution from Midokura favored by DevOps cultures in web scale enterprises and service providers around the world. This session will present case studies from several end user deployments, showing how they use MidoNet to build, run and manage large-scale virtual networks in OpenStack clouds. The session will also discuss how transitioning from a public to private cloud enables organizations to accomplish much more with the same resources, without over-simplifying the inherent complexity of running an OpenStack cloud.
There is a growing trend today of enterprises leveraging both Amazon Web Services (AWS) and on-premise OpenStack-based private clouds. However, the default networking option in OpenStack remains broken and the plethora of confusing plug-ins makes networking in OpenStack mysterious and difficult to manage.
Enter MidoNet, the open source network virtualization solution from Midokura favored by DevOps cultures in web scale enterprises and service providers around the world. This session will present case studies from several end user deployments, showing how they use MidoNet to build, run and manage large-scale virtual networks in OpenStack clouds. The session will also discuss how transitioning from a public to private cloud enables organizations to accomplish much more with the same resources, without over-simplifying the inherent complexity of running an OpenStack cloud.
SaltConf14 - Brendan Burns, Google - Management at Google ScaleSaltStack
As a leading developer of highly scalable, large-scale Web services, Google was forced early on to develop systems to support the deployment and management of diverse workloads at an immense scale. As the broader developer community embraces cloud technologies we see significant parallels between the internal management infrastructure which Google has built over the last decade, and open source management technologies of today. This talk will describe Google's experience in managing large-scale compute services, draw parallels to open source efforts underway today, and sketch out how our past experience shapes our future development of the Google Cloud Platform.
Performance Optimization of Cloud Based Applications by Peter Smith, ACLTriNimbus
Peter Smith, PhD, Principal Software Engineer at ACL talks about Performance Optimization of Cloud Based Applications at TriNimbus' 2017 Canadian Executive Cloud & DevOps summit in Vancouver
August 2018 version of my "What does rename() do", includes the full details on what the Hadoop MapReduce and Spark commit protocols are, so the audience will really understand why rename really, really matters
Put is the new rename: San Jose Summit EditionSteve Loughran
This is the June 2018 variant of the "Put is the new Rename Talk", looking at Hadoop stack integration with object stores, including S3, Azure storage and GCS.
The lessons from implementing a twitter bot designed to live on a raspberry pi and heckle politicians —and deployed into production in the 2017 UK General Election
A review of the state of cloud store integration with the Hadoop stack in 2018; including S3Guard, the new S3A committers and S3 Select.
Presented at Dataworks Summit Berlin 2018, where the demos were live.
Berlin Buzzwords 2017 talk: A look at what our storage models, metaphors and APIs are, showing how we need to rethink the Posix APIs to work with object stores, while looking at different alternatives for local NVM.
This is the unabridged talk; the BBuzz talk was 20 minutes including demo and questions, so had ~half as many slides
Dancing Elephants: Working with Object Storage in Apache Spark and HiveSteve Loughran
A talk looking at the intricate details of working with an object store from Hadoop, Hive, Spark, etc, why the "filesystem" metaphor falls down, and what work myself and others have been up to to try and fix things
Apache Spark and Object Stores —for London Spark User GroupSteve Loughran
The March 2017 version of the "Apache Spark and Object Stores", includes coverage of the Staging Committer. If you'd been at the talk you'd have seen the projector fail just before the demo. It worked earlier! Honest!
My talk from Berlin Buzzwords 2016, looking at whether it is actually possible to lock down a household to the extent you could call it "secure". I also try to highlight that we need to consider "privacy" alongside security.
Hadoop and Kerberos: the Madness Beyond the Gate: January 2016 editionSteve Loughran
An update of the "Hadoop and Kerberos: the Madness Beyond the Gate" talk, covering recent work "the Fix Kerberos" JIRA and its first deliverable: KDiag
2. 22
Small vs Large Clusters
Small Production Clusters and
Proof of Concept
– Build and run by a few skilful
people
– Can be a natural extension
to conventional IT
– You know the servers by
name
Large Production Clusters
– Build and run by pioneers
– Large development staff
– Major Hadoop contributors
– Understand the problems of
scale
Images: Creative Commons 2.0 – Attribution Andrew Morrell (Flickr )
3. 33
– Have, or want to start with, a small PoC (10’s of nodes)
– Want to quickly scale to large cluster (100’s of nodes)
– Want the scale of large clusters, but with the build and operational
model of a small one
– Want to run the cluster rather than build and develop it
– Need to integrate it with existing systems
Large Scale Early Adopters
Unfortunately not all things in life scale as well as Hadoop
Design – The Technology Challenge
Build – The Engineering Challenge
Transfer to Operations - The Service Management Challenge
4. 44
Design – The Technology Challenge
Selecting all the right bits
Server Selection
– Core Nodes: Resilient, Big Memory, RAID
– Data Nodes: Not resilient, no RAID or hot swap, basic iLO
– Trade off Disks vs Cores vs Memory to match target load
– Need to consider disc allocation policy
– Network redundancy is useful to avoid rack switch failures
– Edge Nodes (Data ingress/egress & Mgmt)
– Higher spec data nodes
– Help provide the “appliance” view of the cluster
– Have Hadoop installed but don’t run as part of the cluster.
– Network Selection
– Dual 1Gb from data nodes to rack switches
– 10Gb from rack switches to core, and from Edge nodes
5. 55
Build – The Engineering Challenge
Do you realise how many cardboard boxes that is ?
Building at the scale of 500+ servers has its own set of problems
• Space and Environment
• Consistency of Build
• Failures during the Build
• Deployment time and the cost of rework
Two things we found very helpful:
Factory Integration Services
Cluster Management Utility
6. 66
Build – HP Factory Integration Services
Reducing risk and time
• Many years experience of building large clusters
• Site inspection
• Build, Configure, Soak Test
• Diagnose and fix DoAs
• Rack and Label
• Asset tagging
• Custom build and set-up
• Pack and Ship
• On-Site build and integration
www.hp.com/go/factoryexpress
Complex solutions ...
... Made simple
7. 77
Build – HP Cluster Management Utility
Rack aware deployment and monitoring
• Proven cluster deployment and management tool
• 11 Years of experience
• Proven with clusters of 3500+ nodes
• Deployment
• Network and power load aware deployment
• Easily extensible
• Kickstart integration
• Monitoring
• Scalable non intrusive monitoring
• Collectl integration
• Administration
• Command Line or GUI
• Cluster wide configuration
www.hp.com/go/cmu
10. 1010
Operate – the organisational challenge
How do we know when its working ?
Clusters are not just large numbers of servers
• At scale it may never be 100% up (like a network)
.... but it can be 100% down (like a server)
• Need to think more in terms of “How healthy is it ?”
• Core nodes are important
• Data nodes much less so – unless they fail in patterns
• Edge nodes – somewhere in between
• Look at HDFS health for replication counts
• Nagios & ganglia
• Collectl / CMU to visualise the cluster
11. 1111
Summary
Key considerations when building a large cluster
• Use a pilot system to establish your server configuration
• Stand on the shoulders of the Pioneers
• Build and test in the factory if you can
• Consistency in the build and configuration is vital
• Cherish the NameNode, protect the Edge Nodes, and develop the
right level of indifference to the Data Nodes
• Practice the key recovery cases
• Match training and support to the service expectations
And remember not all things in life scale as well as Hadoop