Practice of large Hadoop cluster in China MobileDataWorks Summit
China Mobile Limited is the leading telecommunications services provider in China, with more than 800 million active users. In China Mobile, distributed big data clusters are built by branch companies in each province for their unique requirements. Meanwhile, we have built a centralized Hadoop cluster with scale more than 1600 nodes, on which we collect data from dozens of distributed clusters and make analysis for our business.
In this session, we will introduce the architecture of the centralized Hadoop cluster and experience of constructing and tuning this large scale Hadoop cluster. Key points are as follows:
1. About Ambari: we improve Ambari with features like supporting HDFS Federation and Ambari HA , improving its performance and enabling it to support up to 1600 nodes.
2. About HDFS: we build a large HDFS cluster with data up to 60PB, using federation, ViewFS, FairCallQueue. Our best practice of cluster operation and management will also be included.
3. About Flume: We use the reformed Flume to collect data as much as 200TB per day.
Speakers
Yuxuan Pan, Software Engineer, China Mobile Software Technology
Duan Yunfeng, Chief Designer of China Mobile's big data system, China Mobile Communications Corporation
Hive LLAP: A High Performance, Cost-effective Alternative to Traditional MPP ...DataWorks Summit
At Walmart Labs, we get close to 200 million customers every week across our 11000+ stores & online all over the world. As part of our data lake initiatives, we started a full-fledged migration to Hadoop based solutions for all our data needs at lower cost than traditional RDBMS/MPP solutions. While we have seen significant success in migrating to Hadoop based Data Lake solutions from traditional RDBMS based data warehouses, one challenge that we have faced is around migrating end users to Hadoop due to query latency issues. To solve this problem and to reduce the cost of the solution, Walmart Labs started using Hive LLAP.
In this session, we will introduce you to Hive LLAP, its architecture, best practices for deployment to achieve sub-second query performance and its cost comparison with traditional RDBMS systems for the same use case.
Data Orchestration Summit 2020 organized by Alluxio
https://www.alluxio.io/data-orchestration-summit-2020/
The Future of Computing is Distributed
Professor Ion Stoica, UC Berkeley RISELab
About Alluxio: alluxio.io
Engage with the open source community on slack: alluxio.io/slack
Practice of large Hadoop cluster in China MobileDataWorks Summit
China Mobile Limited is the leading telecommunications services provider in China, with more than 800 million active users. In China Mobile, distributed big data clusters are built by branch companies in each province for their unique requirements. Meanwhile, we have built a centralized Hadoop cluster with scale more than 1600 nodes, on which we collect data from dozens of distributed clusters and make analysis for our business.
In this session, we will introduce the architecture of the centralized Hadoop cluster and experience of constructing and tuning this large scale Hadoop cluster. Key points are as follows:
1. About Ambari: we improve Ambari with features like supporting HDFS Federation and Ambari HA , improving its performance and enabling it to support up to 1600 nodes.
2. About HDFS: we build a large HDFS cluster with data up to 60PB, using federation, ViewFS, FairCallQueue. Our best practice of cluster operation and management will also be included.
3. About Flume: We use the reformed Flume to collect data as much as 200TB per day.
Speakers
Yuxuan Pan, Software Engineer, China Mobile Software Technology
Duan Yunfeng, Chief Designer of China Mobile's big data system, China Mobile Communications Corporation
Hive LLAP: A High Performance, Cost-effective Alternative to Traditional MPP ...DataWorks Summit
At Walmart Labs, we get close to 200 million customers every week across our 11000+ stores & online all over the world. As part of our data lake initiatives, we started a full-fledged migration to Hadoop based solutions for all our data needs at lower cost than traditional RDBMS/MPP solutions. While we have seen significant success in migrating to Hadoop based Data Lake solutions from traditional RDBMS based data warehouses, one challenge that we have faced is around migrating end users to Hadoop due to query latency issues. To solve this problem and to reduce the cost of the solution, Walmart Labs started using Hive LLAP.
In this session, we will introduce you to Hive LLAP, its architecture, best practices for deployment to achieve sub-second query performance and its cost comparison with traditional RDBMS systems for the same use case.
Data Orchestration Summit 2020 organized by Alluxio
https://www.alluxio.io/data-orchestration-summit-2020/
The Future of Computing is Distributed
Professor Ion Stoica, UC Berkeley RISELab
About Alluxio: alluxio.io
Engage with the open source community on slack: alluxio.io/slack
Elastify Cloud-Native Spark Application with Persistent MemoryDatabricks
Cloud native deployment has become one of the major trends for large scale Big Data analytics. Compared to on-premise data center, cloud offers much stronger scalability and higher elasticity to Big Data applications. However, cloud is also considered to be less performance than on-premise alternatives due to virtualization and cluster resource disaggregation. We present a new cloud native Spark application architecture backed by persistent memory technology. The key ingredient of this architecture is a novel acceleration engine that uses Intel's 3DXPoint technology as external memory. We discuss how the performance of multiple aspects of data processing can be improved using this new architecture. As a key takeaway, audience will gain understanding on the benefits of latest persistent memory technology, and how such new technology could be leveraged in cloud data processing architecture.
High Performance Data Lake with Apache Hudi and Alluxio at T3GoAlluxio, Inc.
Data Orchestration Summit 2020 organized by Alluxio
https://www.alluxio.io/data-orchestration-summit-2020/
High Performance Data Lake with Apache Hudi and Alluxio at T3Go
Trevor Zhang & Vino Yang (T3Go)
About Alluxio: alluxio.io
Engage with the open source community on slack: alluxio.io/slack
Apache Spark 2.4 Bridges the Gap Between Big Data and Deep LearningDataWorks Summit
Big data and AI are joined at the hip: AI applications require massive amounts of training data to build state-of-the-art models. The problem is, big data frameworks like Apache Spark and distributed deep learning frameworks like TensorFlow don’t play well together due to the disparity between how big data jobs are executed and how deep learning jobs are executed.
Apache Spark 2.4 introduced a new scheduling primitive: barrier scheduling. User can indicate Spark whether it should be using the MapReduce mode or barrier mode at each stage of the pipeline, thus it’s easy to embed distributed deep learning training as a Spark stage to simplify the training workflow. In this talk, I will demonstrate how to build a real case pipeline which combines data processing with Spark and deep learning training with TensorFlow step by step. I will also share the best practices and hands-on experiences to show the power of this new features, and bring more discussion on this topic.
Bringing Real-Time to the Enterprise with Hortonworks DataFlowDataWorks Summit
TELUS is Canada’s fastest-growing national telecommunications company, with $12.9B of annual revenue and 12.7M customer connections. TELUS provides TELUS TV to more than 1.1 million customers in western Canada, with the goal of maximizing clients’ service experience. Every television set-top box (STB) in the TELUS network regularly logs its diagnostic state and those logs serve as an important signal of the state of the customers equipment. TELUS’ goal was to move from analysis of these STB logs from a daily, batch process towards streaming analytics that would allow the company to respond within minutes to changes in a device or service health. TELUS increased logging frequency by a factor of 50 and adapted the overall architecture to keep pace with that transformation. Members of the TELUS team will walk through the implementation and workflow challenges they overcame, working with Hortonworks to connect Apache Hadoop, Apache NiFi and Apache Spark, within a secure enterprise environment. They will also discuss the impact to their business and their customers’ satisfaction.
Microsoft has embraced OSS by placing a big bet on Apache YARN to govern the resources of our computing clusters, and we did so by working with the community and adding many new capabilities in YARN. We now look to undertake a similar journey and build the next generation of our job execution engine on top of Apache Tez. We will be building a common platform for executing batch, interactive, ML, and streaming queries at exabyte scale for Microsoft's BigData system called Cosmos. This requires us to push the limits of Tez API to support new graph models, change the executing DAG by dynamically adding new vertices, scheduling for interactive and streaming workloads, squeeze out all the computing power in the cluster by integrating Tez with opportunistic containers in YARN, and scaling a DAG across tens of thousands of machines. We have started out on this journey and want to share our progress, lessons learned, seek help from the community to add these new capabilities, and push Apache Tez to new levels.
SPEAKERS
Hitesh Sharma, Principal Software Engineering Manager, Microsoft Engineering manager in the Big Data team at Microsoft.
Anupam, Senior Software Engineer, Microsoft
Xiaomi is a Chinese technology company, it sells more than 100 million smartphones worldwide in 2018, and also owns one of the world's largest IoT device platforms. Xiaomi builds dozens of mobile apps and Internet services based on intelligent devices, including Ads, news feeds, finance service, game, music, video, personal cloud service and so on. The rapid growth of business results in exponential growth of the data analytics infrastructure. The amount of data has roared more than 20 times in the past 3 years, which renders us big challenges on the HDFS scalability
In this talk, we introduce how we scale HDFS to support hundreds of PB data with thousands nodes:
1. How Xiaomi use Hadoop and the characteristic of our usage
2. We made HDFS federation cluster to be used like a single cluster, most applications don't need to change any code to migrate from a single cluster to a federation cluster. Our works include a wrapper FileSystem compatible with DistributedFileSystem, supporting rename among different name spaces and zookeeper-based mount table renewer.
3. Experience of tuning NameNode to improve scalability
4. How to maintain hundreds of HDFS clusters and the optimization we did on client-side to make user and programs access these clusters easily with high performance
Accelerating Data Computation on Ceph ObjectsAlluxio, Inc.
Alluxio Global Online Meetup
November 10, 2020
For more Alluxio events: https://www.alluxio.io/events/
Speaker(s):
Leonardo Militano, ZHAW
In most of the distributed storage systems, the data nodes are decoupled from compute nodes. This is motivated by an improved cost efficiency, storage utilization and a mutually independent scalability of computation and storage. While this consideration is indisputable, several situations exist where moving computation close to the data brings important benefits. Whenever the stored data is to be processed for analytics purposes, all the data needs to be repeatedly moved from the storage to the compute cluster, which leads to reduced performance.
In this talk, we will present how using Alluxio computation and storage ecosystems can better interact benefiting the "bringing the data close to the code" approach. Moving away from the complete disaggregation of computation and storage, data locality can enhance the computation performance. During this talk, we will present our observations and testing results that will show important enhancements in accelerating Spark Data Analytics on Ceph Objects Storage using Alluxio.
InfluxEnterprise Architectural Patterns by Dean Sheehan, Senior Director, Pre...InfluxData
Dean discusses architecture patterns with InfluxDB Enterprise, covering an overview of InfluxDB Enterprise, features, ingestion and query rates, deployment examples, replication patterns, and general advice.
Accelerate Analytics and ML in the Hybrid Cloud EraAlluxio, Inc.
Alluxio Webinar
April 6, 2021
For more Alluxio events: https://www.alluxio.io/events/
Speakers:
Alex Ma, Alluxio
Peter Behrakis, Alluxio
Many companies we talk to have on premises data lakes and use the cloud(s) to burst compute. Many are now establishing new object data lakes as well. As a result, running analytics such as Hive, Spark, Presto and machine learning are experiencing sluggish response times with data and compute in multiple locations. We also know there is an immense and growing data management burden to support these workflows.
In this talk, we will walk through what Alluxio’s Data Orchestration for the hybrid cloud era is and how it solves the performance and data management challenges we see.
In this tech talk, we'll go over:
- What is Alluxio Data Orchestration?
- How does it work?
- Alluxio customer results
Real-world Cloud HPC at Scale, for Production Workloads (BDT212) | AWS re:Inv...Amazon Web Services
"Running high-performance scientific and engineering applications is challenging no matter where you do it. Join IT executives from Hitachi Global Storage Technology, The Aerospace Corporation, Novartis, and Cycle Computing and learn how they have used the AWS cloud to deploy mission-critical HPC workloads.
Cycle Computing leads the session on how organizations of any scale can run HPC workloads on AWS. Hitachi Global Storage Technology discusses experiences using the cloud to create next-generation hard drives. The Aerospace Corporation provides perspectives on running MPI and other simulations, and offer insights into considerations like security while running rocket science on the cloud. Novartis Institutes for Biomedical Research talks about a scientific computing environment to do performance benchmark workloads and large HPC clusters, including a 30,000-core environment for research in the fight against cancer, using the Cancer Genome Atlas (TCGA)."
Modernizing Global Shared Data Analytics Platform and our Alluxio JourneyAlluxio, Inc.
Data Orchestration Summit 2020 organized by Alluxio
https://www.alluxio.io/data-orchestration-summit-2020/
Modernizing Global Shared Data Analytics Platform and our Alluxio Journey
Sandipan Chakraborty, Director of Engineering (Rakuten)
About Alluxio: alluxio.io
Engage with the open source community on slack: alluxio.io/slack
Elastify Cloud-Native Spark Application with Persistent MemoryDatabricks
Cloud native deployment has become one of the major trends for large scale Big Data analytics. Compared to on-premise data center, cloud offers much stronger scalability and higher elasticity to Big Data applications. However, cloud is also considered to be less performance than on-premise alternatives due to virtualization and cluster resource disaggregation. We present a new cloud native Spark application architecture backed by persistent memory technology. The key ingredient of this architecture is a novel acceleration engine that uses Intel's 3DXPoint technology as external memory. We discuss how the performance of multiple aspects of data processing can be improved using this new architecture. As a key takeaway, audience will gain understanding on the benefits of latest persistent memory technology, and how such new technology could be leveraged in cloud data processing architecture.
High Performance Data Lake with Apache Hudi and Alluxio at T3GoAlluxio, Inc.
Data Orchestration Summit 2020 organized by Alluxio
https://www.alluxio.io/data-orchestration-summit-2020/
High Performance Data Lake with Apache Hudi and Alluxio at T3Go
Trevor Zhang & Vino Yang (T3Go)
About Alluxio: alluxio.io
Engage with the open source community on slack: alluxio.io/slack
Apache Spark 2.4 Bridges the Gap Between Big Data and Deep LearningDataWorks Summit
Big data and AI are joined at the hip: AI applications require massive amounts of training data to build state-of-the-art models. The problem is, big data frameworks like Apache Spark and distributed deep learning frameworks like TensorFlow don’t play well together due to the disparity between how big data jobs are executed and how deep learning jobs are executed.
Apache Spark 2.4 introduced a new scheduling primitive: barrier scheduling. User can indicate Spark whether it should be using the MapReduce mode or barrier mode at each stage of the pipeline, thus it’s easy to embed distributed deep learning training as a Spark stage to simplify the training workflow. In this talk, I will demonstrate how to build a real case pipeline which combines data processing with Spark and deep learning training with TensorFlow step by step. I will also share the best practices and hands-on experiences to show the power of this new features, and bring more discussion on this topic.
Bringing Real-Time to the Enterprise with Hortonworks DataFlowDataWorks Summit
TELUS is Canada’s fastest-growing national telecommunications company, with $12.9B of annual revenue and 12.7M customer connections. TELUS provides TELUS TV to more than 1.1 million customers in western Canada, with the goal of maximizing clients’ service experience. Every television set-top box (STB) in the TELUS network regularly logs its diagnostic state and those logs serve as an important signal of the state of the customers equipment. TELUS’ goal was to move from analysis of these STB logs from a daily, batch process towards streaming analytics that would allow the company to respond within minutes to changes in a device or service health. TELUS increased logging frequency by a factor of 50 and adapted the overall architecture to keep pace with that transformation. Members of the TELUS team will walk through the implementation and workflow challenges they overcame, working with Hortonworks to connect Apache Hadoop, Apache NiFi and Apache Spark, within a secure enterprise environment. They will also discuss the impact to their business and their customers’ satisfaction.
Microsoft has embraced OSS by placing a big bet on Apache YARN to govern the resources of our computing clusters, and we did so by working with the community and adding many new capabilities in YARN. We now look to undertake a similar journey and build the next generation of our job execution engine on top of Apache Tez. We will be building a common platform for executing batch, interactive, ML, and streaming queries at exabyte scale for Microsoft's BigData system called Cosmos. This requires us to push the limits of Tez API to support new graph models, change the executing DAG by dynamically adding new vertices, scheduling for interactive and streaming workloads, squeeze out all the computing power in the cluster by integrating Tez with opportunistic containers in YARN, and scaling a DAG across tens of thousands of machines. We have started out on this journey and want to share our progress, lessons learned, seek help from the community to add these new capabilities, and push Apache Tez to new levels.
SPEAKERS
Hitesh Sharma, Principal Software Engineering Manager, Microsoft Engineering manager in the Big Data team at Microsoft.
Anupam, Senior Software Engineer, Microsoft
Xiaomi is a Chinese technology company, it sells more than 100 million smartphones worldwide in 2018, and also owns one of the world's largest IoT device platforms. Xiaomi builds dozens of mobile apps and Internet services based on intelligent devices, including Ads, news feeds, finance service, game, music, video, personal cloud service and so on. The rapid growth of business results in exponential growth of the data analytics infrastructure. The amount of data has roared more than 20 times in the past 3 years, which renders us big challenges on the HDFS scalability
In this talk, we introduce how we scale HDFS to support hundreds of PB data with thousands nodes:
1. How Xiaomi use Hadoop and the characteristic of our usage
2. We made HDFS federation cluster to be used like a single cluster, most applications don't need to change any code to migrate from a single cluster to a federation cluster. Our works include a wrapper FileSystem compatible with DistributedFileSystem, supporting rename among different name spaces and zookeeper-based mount table renewer.
3. Experience of tuning NameNode to improve scalability
4. How to maintain hundreds of HDFS clusters and the optimization we did on client-side to make user and programs access these clusters easily with high performance
Accelerating Data Computation on Ceph ObjectsAlluxio, Inc.
Alluxio Global Online Meetup
November 10, 2020
For more Alluxio events: https://www.alluxio.io/events/
Speaker(s):
Leonardo Militano, ZHAW
In most of the distributed storage systems, the data nodes are decoupled from compute nodes. This is motivated by an improved cost efficiency, storage utilization and a mutually independent scalability of computation and storage. While this consideration is indisputable, several situations exist where moving computation close to the data brings important benefits. Whenever the stored data is to be processed for analytics purposes, all the data needs to be repeatedly moved from the storage to the compute cluster, which leads to reduced performance.
In this talk, we will present how using Alluxio computation and storage ecosystems can better interact benefiting the "bringing the data close to the code" approach. Moving away from the complete disaggregation of computation and storage, data locality can enhance the computation performance. During this talk, we will present our observations and testing results that will show important enhancements in accelerating Spark Data Analytics on Ceph Objects Storage using Alluxio.
InfluxEnterprise Architectural Patterns by Dean Sheehan, Senior Director, Pre...InfluxData
Dean discusses architecture patterns with InfluxDB Enterprise, covering an overview of InfluxDB Enterprise, features, ingestion and query rates, deployment examples, replication patterns, and general advice.
Accelerate Analytics and ML in the Hybrid Cloud EraAlluxio, Inc.
Alluxio Webinar
April 6, 2021
For more Alluxio events: https://www.alluxio.io/events/
Speakers:
Alex Ma, Alluxio
Peter Behrakis, Alluxio
Many companies we talk to have on premises data lakes and use the cloud(s) to burst compute. Many are now establishing new object data lakes as well. As a result, running analytics such as Hive, Spark, Presto and machine learning are experiencing sluggish response times with data and compute in multiple locations. We also know there is an immense and growing data management burden to support these workflows.
In this talk, we will walk through what Alluxio’s Data Orchestration for the hybrid cloud era is and how it solves the performance and data management challenges we see.
In this tech talk, we'll go over:
- What is Alluxio Data Orchestration?
- How does it work?
- Alluxio customer results
Real-world Cloud HPC at Scale, for Production Workloads (BDT212) | AWS re:Inv...Amazon Web Services
"Running high-performance scientific and engineering applications is challenging no matter where you do it. Join IT executives from Hitachi Global Storage Technology, The Aerospace Corporation, Novartis, and Cycle Computing and learn how they have used the AWS cloud to deploy mission-critical HPC workloads.
Cycle Computing leads the session on how organizations of any scale can run HPC workloads on AWS. Hitachi Global Storage Technology discusses experiences using the cloud to create next-generation hard drives. The Aerospace Corporation provides perspectives on running MPI and other simulations, and offer insights into considerations like security while running rocket science on the cloud. Novartis Institutes for Biomedical Research talks about a scientific computing environment to do performance benchmark workloads and large HPC clusters, including a 30,000-core environment for research in the fight against cancer, using the Cancer Genome Atlas (TCGA)."
Modernizing Global Shared Data Analytics Platform and our Alluxio JourneyAlluxio, Inc.
Data Orchestration Summit 2020 organized by Alluxio
https://www.alluxio.io/data-orchestration-summit-2020/
Modernizing Global Shared Data Analytics Platform and our Alluxio Journey
Sandipan Chakraborty, Director of Engineering (Rakuten)
About Alluxio: alluxio.io
Engage with the open source community on slack: alluxio.io/slack
Run OPNFV Danube on ODCC Scorpio Multi-node Server - Open Software on Open Ha...OPNFV
Zhiqiang Yu, China Mobile, Huabin Tang, China Mobile
Open Data Center Committee (ODCC) is co-founded by Baidu, Tencent, Alibaba, China Telecom, China Mobile, Intel, China Academy of Information and Communications Technology (CAICT). It is a non-profit industrial organization, focusing on researching open hardware such as server, data center and open network technologies to meet the growing demand on hardware in Chinese market.
Scorpio Multi-node Server is an ODCC project sponsored by China Mobile. It is a 4U size server chassis with 8 compute nodes or 4 storage nodes in maximum. It can also be mixture of different kind of servers, like 4 compute nodes and 2 storage nodes. Compared with traditional ATCA or Blade server, Multi-node Server's advantages include:
1.It is easier and cheaper to extend.
2.It has more choices for compute and storage nodes combination.
3.It is easier to maintain by engineers.
4. Even higher density.
5. 4U is more flexible than 10~14U Blade server.
OPNFV develops an integrated and tested open source platform that can be used to build NFV functionality. We are running OPNFV releases Colorado on Scorpio Multi-node smoothly and will try recent Danube on it in China Mobile’s Novonet (Next Generation Network) laboratory.
This presentation will introduce how OPNFV and Scorpio Multi-node server fit perfectly together. It's a fully open implementation of open software and open hardware.
New Ceph capabilities and Reference ArchitecturesKamesh Pemmaraju
Have you heard about Inktank Ceph and are interested to learn some tips and tricks for getting started quickly and efficiently with Ceph? Then this is the session for you!
In this two part session you learn details of:
• the very latest enhancements and capabilities delivered in Inktank Ceph Enterprise such as a new erasure coded storage back-end, support for tiering, and the introduction of user quotas.
• best practices, lessons learned and architecture considerations founded in real customer deployments of Dell and Inktank Ceph solutions that will help accelerate your Ceph deployment.
Software Defined Storage, Big Data and Ceph - What Is all the Fuss About?Red_Hat_Storage
Software Defined Storage, Big Data and Ceph - What Is all the Fuss About? By: Kamesh Pemmaraju,Neil Levine
Have you heard about Inktank Ceph and are interested to learn some tips and tricks for getting started quickly and efficiently with Ceph? Then this is the session for you! In this two part session you learn details of: • the very latest enhancements and capabilities delivered in Inktank Ceph Enterprise such as a new erasure coded storage back-end, support for tiering, and the introduction of user quotas. • best practices, lessons learned and architecture considerations founded in real customer deployments of Dell and Inktank Ceph solutions that will help accelerate your Ceph deployment.
OpenStack at the speed of business with SolidFire & Red Hat NetApp
When it comes to OpenStack® and the enterprise, it’s critical that you can rapidly deploy a plug-and-play solution that delivers mixed workload capabilities on a shared infrastructure. Join Red Hat and SolidFire to see how Agile Infrastructure for OpenStack can help your cloud move at the speed of business.
Optimising Service Deployment and Infrastructure Resource ConfigurationRECAP Project
This is a presentation delivered by Alec Leckey (Intel) at the 2nd Data Centre Symposium held in conjunction with the National Conference on Cloud Computing and Commerce (http://2018.nc4.ie/) on April 10, 2018 in Dublin, Ireland.
Learn more about the RECAP project: https://recap-project.eu/
Install the Intel Landscaper: https://github.com/IntelLabsEurope/landscaper
Capital One Delivers Risk Insights in Real Time with Stream Processingconfluent
Speakers: Ravi Dubey, Senior Manager, Software Engineering, Capital One + Jeff Sharpe, Software Engineer, Capital One
Capital One supports interactions with real-time streaming transactional data using Apache Kafka®. Kafka helps deliver information to internal operation teams and bank tellers to assist with assessing risk and protect customers in a myriad of ways.
Inside the bank, Kafka allows Capital One to build a real-time system that takes advantage of modern data and cloud technologies without exposing customers to unnecessary data breaches, or violating privacy regulations. These examples demonstrate how a streaming platform enables Capital One to act on their visions faster and in a more scalable way through the Kafka solution, helping establish Capital One as an innovator in the banking space.
Join us for this online talk on lessons learned, best practices and technical patterns of Capital One’s deployment of Apache Kafka.
-Find out how Kafka delivers on a 5-second service-level agreement (SLA) for inside branch tellers.
-Learn how to combine and host data in-memory and prevent personally identifiable information (PII) violations of in-flight transactions.
-Understand how Capital One manages Kafka Docker containers using Kubernetes.
Watch the recording: https://videos.confluent.io/watch/6e6ukQNnmASwkf9Gkdhh69?.
OSMC 2019 | Monitoring Alerts and Metrics on Large Power Systems Clusters by ...NETWAYS
In this talk we’ll introduce an open source project being used to monitor large Power Systems clusters, such as in the IBM collaboration with Oak Ridge and Lawrence Livermore laboratories for the Summit project, a large deployment of custom AC922 Power Systems nodes augmented by GPUs that work in tandem to implement the (currently) largest Supercomputer in the world.
Data is collected out-of-band directly from the firmware layer and then redistributed to various components using an open source component called Crassd. In addition, in-band operating-system and service level metrics, logs and alerts can also be collected and used to enrich the visualization dashboards. Open source components such as the Elastic Stack (Elasticsearch, Logstash, Kibana and select Beats) and Netdata are used for monitoring scenarios appropriate to each tool’s strengths, with other components such as Prometheus and Grafana in the process of being implemented. We’ll briefly discuss our experience to put these components together, and the decisions we had to make in order to automate their deployment and configuration for our goals. Finally, we lay out collaboration possibilities and future directions to enhance our project as a convenient starting point for others in the open source community to easily monitor their own Power Systems environments.
Application and Network Orchestration using Heat & ToscaNati Shalom
The buzzwords Neutron, Heat, and TOSCA are spoken about quite often when it comes to the OpenStack - and many of us are still trying to make sense of the terminology and its place in the OpenStack world.
Where OpenStack Neutron provides APIs for creating network elements, OpenStack Heat provides an orchestration engine for automating the setup and configuration of OpenStack infrastructure, while TOSCA is a standard for templating and defining application topology and policies (that form the basis for Heat). In this context, it really makes sense to put these all together to achieve application and network automation for OpenStack on steroids.
In this session we will learn how we can use the robust combination of Heat and TOSCA to configure and control resources on Nova and Neutron in order to automate the network configuration as part of the application deployment.
The session will include a demo and code examples that show how you can configure virtual networks, attach public IPs, set up security groups, set up load balancing and automatically scale up/down and more. You will leave this session understanding where Neutron meets Heat and TOSCA.
This talk was delivered as part of OpenStack Paris summit - 2014 - http://openstacksummitnovember2014paris.sched.org/event/2b85b682ccaf3a5961e463b61e2403f8#.VFeuG_TF8mc
An introduction to the key concepts of SDN and NFV with visuals of:
- How SDN is transforming the Data Center
- How NFV is transforming the Service Provider domain and the End-customer domain
- Objectives
- Origin
- Ambassadors
- Applicability
- Analogies
- Benefits
- Industry Standards
- Drivers
- Obstacles
- Growth
- Resources and Events
Design and build a Private Cloud for your Enterprise using a Scalable Architecture.
- Bridge IT and the Public Cloud
- Reduce Cost
- On-Demand Services
- Run Scalable Applications
- Handle Traffic Growth
- Meet Compliance Objectives
- Offer Operational Flexibility and Efficiency
VMworld 2013: How SRP Delivers More Than Power to Their Customers VMworld
VMworld 2013
Sheldon Brown, SRP
Girish Manmadkar, VMware
Learn more about VMworld and register at http://www.vmworld.com/index.jspa?src=socmed-vmworld-slideshare
Madhu Rangarajan will provide an overview of Networking trends they are seeing in Cloud, various network topologies and tradeoffs, and trends in the acceleration of packet processing workloads. They will also talk about some of the work going on in Intel to address these trends, including FPGAs in the datacenter.
Similar to Optimized placement in Openstack for NFV (20)
Donabe is a multi-tier Application Container Service and a top level orchestrator that will provision/deploy complex apps. These slides were presented at the Openstack Essex Design Summit in Boston.
Instagram has become one of the most popular social media platforms, allowing people to share photos, videos, and stories with their followers. Sometimes, though, you might want to view someone's story without them knowing.
Bridging the Digital Gap Brad Spiegel Macon, GA Initiative.pptxBrad Spiegel Macon GA
Brad Spiegel Macon GA’s journey exemplifies the profound impact that one individual can have on their community. Through his unwavering dedication to digital inclusion, he’s not only bridging the gap in Macon but also setting an example for others to follow.
2.Cellular Networks_The final stage of connectivity is achieved by segmenting...JeyaPerumal1
A cellular network, frequently referred to as a mobile network, is a type of communication system that enables wireless communication between mobile devices. The final stage of connectivity is achieved by segmenting the comprehensive service area into several compact zones, each called a cell.
Understanding User Behavior with Google Analytics.pdfSEO Article Boost
Unlocking the full potential of Google Analytics is crucial for understanding and optimizing your website’s performance. This guide dives deep into the essential aspects of Google Analytics, from analyzing traffic sources to understanding user demographics and tracking user engagement.
Traffic Sources Analysis:
Discover where your website traffic originates. By examining the Acquisition section, you can identify whether visitors come from organic search, paid campaigns, direct visits, social media, or referral links. This knowledge helps in refining marketing strategies and optimizing resource allocation.
User Demographics Insights:
Gain a comprehensive view of your audience by exploring demographic data in the Audience section. Understand age, gender, and interests to tailor your marketing strategies effectively. Leverage this information to create personalized content and improve user engagement and conversion rates.
Tracking User Engagement:
Learn how to measure user interaction with your site through key metrics like bounce rate, average session duration, and pages per session. Enhance user experience by analyzing engagement metrics and implementing strategies to keep visitors engaged.
Conversion Rate Optimization:
Understand the importance of conversion rates and how to track them using Google Analytics. Set up Goals, analyze conversion funnels, segment your audience, and employ A/B testing to optimize your website for higher conversions. Utilize ecommerce tracking and multi-channel funnels for a detailed view of your sales performance and marketing channel contributions.
Custom Reports and Dashboards:
Create custom reports and dashboards to visualize and interpret data relevant to your business goals. Use advanced filters, segments, and visualization options to gain deeper insights. Incorporate custom dimensions and metrics for tailored data analysis. Integrate external data sources to enrich your analytics and make well-informed decisions.
This guide is designed to help you harness the power of Google Analytics for making data-driven decisions that enhance website performance and achieve your digital marketing objectives. Whether you are looking to improve SEO, refine your social media strategy, or boost conversion rates, understanding and utilizing Google Analytics is essential for your success.
APNIC Foundation, presented by Ellisha Heppner at the PNG DNS Forum 2024APNIC
Ellisha Heppner, Grant Management Lead, presented an update on APNIC Foundation to the PNG DNS Forum held from 6 to 10 May, 2024 in Port Moresby, Papua New Guinea.
Gen Z and the marketplaces - let's translate their needsLaura Szabó
The product workshop focused on exploring the requirements of Generation Z in relation to marketplace dynamics. We delved into their specific needs, examined the specifics in their shopping preferences, and analyzed their preferred methods for accessing information and making purchases within a marketplace. Through the study of real-life cases , we tried to gain valuable insights into enhancing the marketplace experience for Generation Z.
The workshop was held on the DMA Conference in Vienna June 2024.
1.Wireless Communication System_Wireless communication is a broad term that i...JeyaPerumal1
Wireless communication involves the transmission of information over a distance without the help of wires, cables or any other forms of electrical conductors.
Wireless communication is a broad term that incorporates all procedures and forms of connecting and communicating between two or more devices using a wireless signal through wireless communication technologies and devices.
Features of Wireless Communication
The evolution of wireless technology has brought many advancements with its effective features.
The transmitted distance can be anywhere between a few meters (for example, a television's remote control) and thousands of kilometers (for example, radio communication).
Wireless communication can be used for cellular telephony, wireless access to the internet, wireless home networking, and so on.
1. Increasing
Infrastructure
Ef/iciency
via
Optimized
NFV
Placement
in
OpenStack
Clouds
Yathiraj
Udupi,
Debo
Dutta
–
Cisco
Ram
(Ramki)
Krishnan
-‐
Brocade
OpenStack
Atlanta
Summit,
May
2014
2. Who and why?
Debo/Yathi
-‐
Cisco
Cloud
CTO
ofLice
Ramki
-‐
Brocade
CTO
ofLice
Goal:
Drive
Innovative
Open
Source
solutions
for
NFV
with
OpenStack
3. Our Thesis
• Toby
Ford@AT&T’s
NFV
talk
on
Tue,
May
13th
• Worlds
of
IT
and
Telco
are
coming
together
• Telco
Cloud
-‐
OpenStack
as
the
infrastructure
foundation
• Goal:
Transform
OpenStack
to
a
Carrier-‐grade
cloud
solution
• We
deep
dive
into
some
high-‐level
gaps
Toby
identiLied
• We
demo
some
initial
progress
4. Agenda
• NFV
Summary
• Cloud
NFV
Use
Case
• Drive
Innovation
-‐
EfLicient
Resource
Placement
Strategies
• Extensions
to
OpenStack
scheduler
• Conclusion
5. Network Functions Virtualization (NFV)
NFV Vision
Source: ETSI NFV White Paper
• Global
movement
by
network
operators
-‐
AT&T,
Verizon,
BT,
CenturyLink,
Deutsche
Telekom,
Telefonica,
KDDI
etc.
• General
purpose
hardware
-‐
OPEX
and
CAPEX
savings
• Increased
automation
–
OPEX
savings,
faster
time
to
market
• New
business
models
and
value
added
services
6. NFV Use Case - NFVIaas
Motivation
• Network
Functions
in
the
cloud
• Combined
value
–
Infrastructure
as
a
service
(IaaS)
–
Compute/storage
infra,
Network
as
a
service
(NaaS)
–
WAN
network
infra
• Leverage
NFV
Infra
of
another
SP
–
increase
resiliency,
reduce
latency
(CDN),
regulatory
requirements
Where
are
we
are
today
?
• Compute/storage
is
treated
independent
of
network,
no
energy
efLiciency
considerations
• Service
value
is
not
maximized
NFV Use Case – NFVIaaS
Source: ETSI NFV Use Cases
7. NaaS
Virtualized
Network
Bandwidth
Bandwidth
Virtual
Machine
Virtual
Machine
Virtual
Machine
Virtual
Machine
WAN Bandwidth on Demand
Data
Center
1
Data
Center
2
BeneLits
• Use
WAN
bandwidth
as
needed,
avoid
Lixed
cost
due
to
reservation
(typically
1.5
times
peak)
–
typically
leverage
MPLS
technologies
• Popular
use
cases
-‐
Disaster
Recovery
,
On-‐demand
backup
across
WAN
StorageStorage
8. NFVIaas (IaaS+NaaS)
Virtualized
Network
Bandwidth
Bandwidth
Virtual
Machine
Virtual
Machine
Virtual
Machine
Virtual
Machine
Data
Center
1
Data
Center
2
Where
do
we
want
to
get
to
?
• Beyond
WAN
Bandwidth
savings
• Optimal
resource
placement
across
DCs
-‐
Increase
Energy
efLiciency
while
maintaining
multi-‐tenant
fairness
and
improving
performance
–
CAPEX/OPEX
savings,
Improve
QoE,
Address
regulatory
requirements
• Popular
use
cases
-‐
Disaster
Recovery,
On-‐demand
backup
across
WAN,
Virtualized
CDN
Compute/Storage/WAN Bandwidth on Demand + Energy Efficiency
Storage Storage
9. NFVIaas (IaaS+NaaS)
• Power
usage
in
DCs
-‐-‐
servers
à
heavy
hitter
• Server
power
proLiles
typically
non-‐linear;
~45%
of
peak
power
with
~20%
of
offered
load;
~30%
power
in
idle
state
• InefLicient
to
keep
servers
powered
on
under
low
load
conditions
Energy Efficiency Issues
SPEC
Benchmark
results:
HP
ProLiant
DL380p
rack
server
Source:
http://i.dell.com/sites/doccontent/shared-‐content/data-‐sheets/en/Documents/Comparing-‐Dell-‐R720-‐and-‐HP-‐Proliant-‐DL380p-‐Gen8-‐Servers.pdf
10. NFV – Huge opportunity for Openstack
Energy
aware
joint
scheduling
of
compute/storage/networking
resources
–
example
below
• NFV
Customer
submits
job
request,
e.g.
backup,
with
elasticity
windows
• NFV
Provider
returns
back
information
about
time
window
to
schedule
backup
• Trigger
other
events
e.g.
Consolidate
workloads;
Finish
one
job
and
start
and
the
next;
Power
down
resources
(especially
servers)
after
job
completion
Our
Solution:
Smart
Scheduler
in
Openstack
How do we get there ?
Solver
Scheduler
Adapted
from
ETSI
NFV
Architectural
Framework
11. Users:
Minimize costs… (Energy &
Network Efficiency)
Maximize Performance...
Infrastructure:
State (BigData?)
(Storage/Network/Compute
state, Energy Profiles, Policy/
constraints etc.)
Smart Scheduling in
Smart Scheduling in OpenStack for Optimized
NFV Resource Placements
Our Solution Smart
Scheduler in Openstack
• Use analytics to determine current state
of the Openstack deployment.
• Use resource management techniques
to pick resources based on business
constraints
12. Candidate Solution: Unified Constraints-based
Scheduling
A Smart Resource Placement Engine
• Unified constraints involving network,
storage, compute, energy, etc.
• Global state + analytics
• Blazing fast implementations via Apache
licensed third-party Solver libraries
Sources:
• https://docs.google.com/document/d/1IiPI0sfaWb1bdYiMWzAAx0HYR6UqzOan_Utgml5W1HI/edit
• https://github.com/CiscoSystems/nova-solver-scheduler
13. Solver Scheduler: Smart Scheduling in OS
Intelligent Placement
Engine
Plug in Plug in
Scheduling
Decision
Cost
Functions
Constraint
Functions
Users:
Minimize costs… (Energy & Network
Efficiency)
Maximize Performance...
Infrastructure:
Server State...
Energy Profiles…
Network Link Capacities…
System Capacity...Sources:
• https://docs.google.com/document/d/1IiPI0sfaWb1bdYiMWzAAx0HYR6UqzOan_Utgml5W1HI/edit
• https://github.com/CiscoSystems/nova-solver-scheduler
14. An Example LP Problem
Formulation
Supply
Demand
Cost
Metric
to
Minimize
Constraints
to
satisfy
Cost
Variables
to
solve
Scheduling can be Complex
15. DEMO: Smart Scheduling for NFV Service VMs with
Compute/Storage Affinity Constraints
Applicable Scenarios:
1. CDN NFV Service VMs that need data on
certain storage volumes, on physical
servers that are on or closest to the data.
2. Backup NFV Service VMs placement.
Multinode devstack setup:
- Host-1: (Controller, Compute node)
- Host-2: (Compute node with demo_vol_1
Volume)
- Host-3: (Compute node with demo_vol_2
Volume)
Boot 2 VMs specifying the requested volumes to
be close in proximity
Results: Optimal placement by picking the
two physical volume hosts: Host-2 and
Host-3.
Host-‐3:
Host-‐2:
Host-‐1:
demo_vol_1
demo_vol_2
17. Conclusion
• NFV
Value
Proposition
• NVF
is
a
killer
use-‐case
for
Openstack
• Call
for
community
action
• Scheduler
Gap
and
a
candidate
solution
[e.g.
SolverScheduler,
blueprint
exists,
code
pushed
for
review
in
Icehouse]
• Cross-‐Scheduler
API
w.
constraints
[e.g.
augment
server-‐groups
API
released
in
Icehouse]
• Neutron
hooks
for
Virtual
Network
Services
(and
API)