Managing Apache Spark Workload and Automatic OptimizingDatabricks
eBay is highly using Spark as one of most significant data engines. In data warehouse domain, there are millions of batch queries running every day against 6000+ key DW tables, which contains over 22PB data (compressed) and still keeps booming every year. In machine learning domain, it is playing a more and more significant role. We have introduced our great achievement in migration work from MPP database to Apache Spark last year in Europe Summit. Furthermore, from the vision of the entire infrastructure, it is still a big challenge on managing workload and efficiency for all Spark jobs upon our data center. Our team is leading the whole infrastructure of big data platform and the management tools upon it, helping our customers -- not only DW engineers and data scientists, but also AI engineers -- to leverage on the same page. In this session, we will introduce how to benefit all of them within a self-service workload management portal/system. First, we will share the basic architecture of this system to illustrate how it collects metrics from multiple data centers and how it detects the abnormal workload real-time. We develop a component called Profiler which is to enhance the current Spark core to support customized metric collection. Next, we will demonstrate some real user stories in eBay to show how the self-service system reduces the efforts both in customer side and infra-team side. That's the highlight part about Spark job analysis and diagnosis. Finally, some incoming advanced features will be introduced to describe an automatic optimizing workflow rather than just alerting.
Speaker: Lantao Jin
Tutorial: Using GoBGP as an IXP connecting routerShu Sugimoto
- Show you how GoBGP can be used as a software router in conjunction with quagga
- (Tutorial) Walk through the setup of IXP connecting router using GoBGP
GitLab Commit: Enhance your Compliance with Policy-Based CI/CDNico Meisenzahl
Whether you want to get started with Governance or improve your current process, this talk will show you how to improve your compliance by implementing policy-based CI/CD (Continuous Integration / Continuous Delivery) with GitLab CI and Open Policy Agent.
Philippe and Nico will tell you all the details about Open Policy Agent and how you can easily integrate it into your existing CI/CD pipelines. Join our session to learn how to improve compliance, from gating your dependencies to controlling your infrastructure.
Airflow: Save Tons of Money by Using Deferrable OperatorsKaxil Naik
This talk is from Open Source Summit 2022
Apache Airflow 2.2 introduced the concept of Deferrable Tasks that uses Python's async feature.
All the Airflow sensors and poll-based operators can be hugely optimized to save tons of money by freeing up worker slots when polling.
This session will cover the following topics: - Introduction to the concept of deferrable operator
- Why do we need them?
- When to use them?
- How does it work?
- Writing Custom deferrable operators & Sensors
Webinar topic: OSPF On Router OS7
Presenter: Achmad Mardiansyah & M. Taufik Nurhuda
In this webinar series, How OSPF On Router OS7
Please share your feedback or webinar ideas here: http://bit.ly/glcfeedback
Check our schedule for future events: https://www.glcnetworks.com/en/schedule/
Follow our social media for updates: Facebook, Instagram, YouTube Channel, and telegram also discord
Recording available on Youtube
https://youtu.be/nuByFdZHvAg
Managing Apache Spark Workload and Automatic OptimizingDatabricks
eBay is highly using Spark as one of most significant data engines. In data warehouse domain, there are millions of batch queries running every day against 6000+ key DW tables, which contains over 22PB data (compressed) and still keeps booming every year. In machine learning domain, it is playing a more and more significant role. We have introduced our great achievement in migration work from MPP database to Apache Spark last year in Europe Summit. Furthermore, from the vision of the entire infrastructure, it is still a big challenge on managing workload and efficiency for all Spark jobs upon our data center. Our team is leading the whole infrastructure of big data platform and the management tools upon it, helping our customers -- not only DW engineers and data scientists, but also AI engineers -- to leverage on the same page. In this session, we will introduce how to benefit all of them within a self-service workload management portal/system. First, we will share the basic architecture of this system to illustrate how it collects metrics from multiple data centers and how it detects the abnormal workload real-time. We develop a component called Profiler which is to enhance the current Spark core to support customized metric collection. Next, we will demonstrate some real user stories in eBay to show how the self-service system reduces the efforts both in customer side and infra-team side. That's the highlight part about Spark job analysis and diagnosis. Finally, some incoming advanced features will be introduced to describe an automatic optimizing workflow rather than just alerting.
Speaker: Lantao Jin
Tutorial: Using GoBGP as an IXP connecting routerShu Sugimoto
- Show you how GoBGP can be used as a software router in conjunction with quagga
- (Tutorial) Walk through the setup of IXP connecting router using GoBGP
GitLab Commit: Enhance your Compliance with Policy-Based CI/CDNico Meisenzahl
Whether you want to get started with Governance or improve your current process, this talk will show you how to improve your compliance by implementing policy-based CI/CD (Continuous Integration / Continuous Delivery) with GitLab CI and Open Policy Agent.
Philippe and Nico will tell you all the details about Open Policy Agent and how you can easily integrate it into your existing CI/CD pipelines. Join our session to learn how to improve compliance, from gating your dependencies to controlling your infrastructure.
Airflow: Save Tons of Money by Using Deferrable OperatorsKaxil Naik
This talk is from Open Source Summit 2022
Apache Airflow 2.2 introduced the concept of Deferrable Tasks that uses Python's async feature.
All the Airflow sensors and poll-based operators can be hugely optimized to save tons of money by freeing up worker slots when polling.
This session will cover the following topics: - Introduction to the concept of deferrable operator
- Why do we need them?
- When to use them?
- How does it work?
- Writing Custom deferrable operators & Sensors
Webinar topic: OSPF On Router OS7
Presenter: Achmad Mardiansyah & M. Taufik Nurhuda
In this webinar series, How OSPF On Router OS7
Please share your feedback or webinar ideas here: http://bit.ly/glcfeedback
Check our schedule for future events: https://www.glcnetworks.com/en/schedule/
Follow our social media for updates: Facebook, Instagram, YouTube Channel, and telegram also discord
Recording available on Youtube
https://youtu.be/nuByFdZHvAg
This presentation discusses the following topics:
Hadoop Distributed File System (HDFS)
How does HDFS work?
HDFS Architecture
Features of HDFS
Benefits of using HDFS
Examples: Target Marketing
HDFS data replication
Webinar topic: IPv6 with Mikrotik
Presenter: Achmad Mardiansyah
In this webinar series, We are discussing IPv6 with Mikrotik
Please share your feedback or webinar ideas here: http://bit.ly/glcfeedback
Check our schedule for future events: https://www.glcnetworks.com/en/
Follow our social media for updates: Facebook, Instagram, YouTube Channel, and telegram
The recording is available On :
https://youtu.be/C8Tfh1a9y20
Real Time UI with Apache Kafka Streaming Analytics of Fast Data and Server PushLucas Jellema
Fast data arrives in real time and potentially high volume. Rapid processing, filtering and aggregation is required to ensure timely reaction and actual information in user interfaces. Doing so is a challenge, make this happen in a scalable and reliable fashion is even more interesting. This session introduces Apache Kafka as the scalable event bus that takes care of the events as they flow in and Kafka Streams for the streaming analytics. Both Java and Node applications are demonstrated that interact with Kafka and leverage Server Sent Events and WebSocket channels to update the Web UI in real time. User activity performed by the audience in the Web UI is processed by the Kafka powered back end and results in live updates on all clients. Kafka Streams and KSQL are used to analyze the real time events in real time and publish events with the live findings.
Webinar topic: Mikrotik Load Balancing with PCC
Presenter: Achmad Mardiansyah
In this webinar series, We are discussing Mikrotik Load Balancing with PCC
Please share your feedback or webinar ideas here: http://bit.ly/glcfeedback
Check our schedule for future events: https://www.glcnetworks.com/schedule/
Follow our social media for updates: Facebook, Instagram, YouTube Channel, and telegram
Recording is available on Youtube
https://youtu.be/3leJgk9u7Gw
Webinar topic: Zabbix for Monitoring
Presenter: Achmad Mardiansyah
In this webinar series, How Zabbix for Monitoring
Please share your feedback or webinar ideas here: http://bit.ly/glcfeedback
Check our schedule for future events: https://www.glcnetworks.com/en/schedule/
Follow our social media for updates: Facebook, Instagram, YouTube Channel, and telegram also discord
Recording available on Youtube
https://youtu.be/iUs2G_9FS-M
Event-Driven Messaging and Actions using Apache Flink and Apache NiFiDataWorks Summit
At Comcast, our team has been architecting a customer experience platform which is able to react to near-real-time events and interactions and deliver appropriate and timely communications to customers. By combining the low latency capabilities of Apache Flink and the dataflow capabilities of Apache NiFi we are able to process events at high volume to trigger, enrich, filter, and act/communicate to enhance customer experiences. Apache Flink and Apache NiFi complement each other with their strengths in event streaming and correlation, state management, command-and-control, parallelism, development methodology, and interoperability with surrounding technologies. We will trace our journey from starting with Apache NiFi over three years ago and our more recent introduction of Apache Flink into our platform stack to handle more complex scenarios. In this presentation we will compare and contrast which business and technical use cases are best suited to which platform and explore different ways to integrate the two platforms into a single solution.
FOSDEM15 SDN developer room talk
DPDK performance
How to not just do a demo with DPDK
The Intel DPDK provides a platform for building high performance Network Function Virtualization applications. But it is hard to get high performance unless certain design tradeoffs are made. This talk focuses on the lessons learned in creating the Brocade vRouter using DPDK. It covers some of the architecture, locking and low level issues that all have to be dealt with to achieve 80 Million packets per second forwarding.
Improving Apache Spark by Taking Advantage of Disaggregated ArchitectureDatabricks
Shuffle in Apache Spark is an intermediate phrase redistributing data across computing units, which has one important primitive that the shuffle data is persisted on local disks. This architecture suffers from some scalability and reliability issues. Moreover, the assumptions of collocated storage do not always hold in today's data centers. The hardware trend is moving to disaggregated storage and compute architecture for better cost efficiency and scalability. To address the issues of Spark shuffle and support disaggregated storage and compute architecture, we implemented a new remote Spark shuffle manager. This new architecture writes shuffle data to a remote cluster with different Hadoop-compatible filesystem backends. Firstly, the failure of compute nodes will no longer cause shuffle data recomputation. Spark executors can also be allocated and recycled dynamically which results in better resource utilization. Secondly, for most customers currently running Spark with collocated storage, it is usually challenging for them to upgrade the disks on every node to latest hardware like NVMe SSD and persistent memory because of cost consideration and system compatibility. With this new shuffle manager, they are free to build a separated cluster storing and serving the shuffle data, leveraging the latest hardware to improve the performance and reliability. Thirdly, in HPC world, more customers are trying Spark as their high performance data analytics tools, while storage and compute in HPC clusters are typically disaggregated. This work will make their life easier. In this talk, we will present an overview of the issues of the current Spark shuffle implementation, the design of new remote shuffle manager, and a performance study of the work.
When you think about C#, you'll usually think about a high-level language, one that is utilized to build websites, APIs, and desktop applications. However, from its inception, C# had the foundation to be used as a system language, with facilities that allow you direct memory access and fine-grained control over memory and execution.
In the last five years, there has been a huge emphasis on making C# a more capable language for system development. Oren Eini, the founder of RavenDB, has used C# as the base language to build a distributed document database for over a decade.
In this talk, Oren will discuss the features that make C# a viable system language for building high-end systems. Learn how you can mix and match, in a single project, both high-level concepts and intimate control over every single thing that is happening in your system.
Analytical Modeling of End-to-End Delay in OpenFlow Based NetworksAzeem Iqbal
OpenFlow enabled networks split and separate the data and control planes of traditional networks. This design commodifies network switches and enables centralized control of the network. Control decisions are made by an OpenFlow controller, and locally cached by switches, as directed by controllers. Since controllers are not necessarily co-located with switches that can significantly impact the forwarding delay incurred by packets in switches. Only very few studies have been conducted to evaluate the performance of OpenFlow in terms of end-to-end delay. In this work we develop a stochastic model for the end to end delay in OpenFlow switches based on measurements made in Internetscale experiments performed on three different platforms, i.e. Mininet, the GENI testbed and the OF@TEIN testbed.
This presentation discusses the following topics:
Hadoop Distributed File System (HDFS)
How does HDFS work?
HDFS Architecture
Features of HDFS
Benefits of using HDFS
Examples: Target Marketing
HDFS data replication
Webinar topic: IPv6 with Mikrotik
Presenter: Achmad Mardiansyah
In this webinar series, We are discussing IPv6 with Mikrotik
Please share your feedback or webinar ideas here: http://bit.ly/glcfeedback
Check our schedule for future events: https://www.glcnetworks.com/en/
Follow our social media for updates: Facebook, Instagram, YouTube Channel, and telegram
The recording is available On :
https://youtu.be/C8Tfh1a9y20
Real Time UI with Apache Kafka Streaming Analytics of Fast Data and Server PushLucas Jellema
Fast data arrives in real time and potentially high volume. Rapid processing, filtering and aggregation is required to ensure timely reaction and actual information in user interfaces. Doing so is a challenge, make this happen in a scalable and reliable fashion is even more interesting. This session introduces Apache Kafka as the scalable event bus that takes care of the events as they flow in and Kafka Streams for the streaming analytics. Both Java and Node applications are demonstrated that interact with Kafka and leverage Server Sent Events and WebSocket channels to update the Web UI in real time. User activity performed by the audience in the Web UI is processed by the Kafka powered back end and results in live updates on all clients. Kafka Streams and KSQL are used to analyze the real time events in real time and publish events with the live findings.
Webinar topic: Mikrotik Load Balancing with PCC
Presenter: Achmad Mardiansyah
In this webinar series, We are discussing Mikrotik Load Balancing with PCC
Please share your feedback or webinar ideas here: http://bit.ly/glcfeedback
Check our schedule for future events: https://www.glcnetworks.com/schedule/
Follow our social media for updates: Facebook, Instagram, YouTube Channel, and telegram
Recording is available on Youtube
https://youtu.be/3leJgk9u7Gw
Webinar topic: Zabbix for Monitoring
Presenter: Achmad Mardiansyah
In this webinar series, How Zabbix for Monitoring
Please share your feedback or webinar ideas here: http://bit.ly/glcfeedback
Check our schedule for future events: https://www.glcnetworks.com/en/schedule/
Follow our social media for updates: Facebook, Instagram, YouTube Channel, and telegram also discord
Recording available on Youtube
https://youtu.be/iUs2G_9FS-M
Event-Driven Messaging and Actions using Apache Flink and Apache NiFiDataWorks Summit
At Comcast, our team has been architecting a customer experience platform which is able to react to near-real-time events and interactions and deliver appropriate and timely communications to customers. By combining the low latency capabilities of Apache Flink and the dataflow capabilities of Apache NiFi we are able to process events at high volume to trigger, enrich, filter, and act/communicate to enhance customer experiences. Apache Flink and Apache NiFi complement each other with their strengths in event streaming and correlation, state management, command-and-control, parallelism, development methodology, and interoperability with surrounding technologies. We will trace our journey from starting with Apache NiFi over three years ago and our more recent introduction of Apache Flink into our platform stack to handle more complex scenarios. In this presentation we will compare and contrast which business and technical use cases are best suited to which platform and explore different ways to integrate the two platforms into a single solution.
FOSDEM15 SDN developer room talk
DPDK performance
How to not just do a demo with DPDK
The Intel DPDK provides a platform for building high performance Network Function Virtualization applications. But it is hard to get high performance unless certain design tradeoffs are made. This talk focuses on the lessons learned in creating the Brocade vRouter using DPDK. It covers some of the architecture, locking and low level issues that all have to be dealt with to achieve 80 Million packets per second forwarding.
Improving Apache Spark by Taking Advantage of Disaggregated ArchitectureDatabricks
Shuffle in Apache Spark is an intermediate phrase redistributing data across computing units, which has one important primitive that the shuffle data is persisted on local disks. This architecture suffers from some scalability and reliability issues. Moreover, the assumptions of collocated storage do not always hold in today's data centers. The hardware trend is moving to disaggregated storage and compute architecture for better cost efficiency and scalability. To address the issues of Spark shuffle and support disaggregated storage and compute architecture, we implemented a new remote Spark shuffle manager. This new architecture writes shuffle data to a remote cluster with different Hadoop-compatible filesystem backends. Firstly, the failure of compute nodes will no longer cause shuffle data recomputation. Spark executors can also be allocated and recycled dynamically which results in better resource utilization. Secondly, for most customers currently running Spark with collocated storage, it is usually challenging for them to upgrade the disks on every node to latest hardware like NVMe SSD and persistent memory because of cost consideration and system compatibility. With this new shuffle manager, they are free to build a separated cluster storing and serving the shuffle data, leveraging the latest hardware to improve the performance and reliability. Thirdly, in HPC world, more customers are trying Spark as their high performance data analytics tools, while storage and compute in HPC clusters are typically disaggregated. This work will make their life easier. In this talk, we will present an overview of the issues of the current Spark shuffle implementation, the design of new remote shuffle manager, and a performance study of the work.
When you think about C#, you'll usually think about a high-level language, one that is utilized to build websites, APIs, and desktop applications. However, from its inception, C# had the foundation to be used as a system language, with facilities that allow you direct memory access and fine-grained control over memory and execution.
In the last five years, there has been a huge emphasis on making C# a more capable language for system development. Oren Eini, the founder of RavenDB, has used C# as the base language to build a distributed document database for over a decade.
In this talk, Oren will discuss the features that make C# a viable system language for building high-end systems. Learn how you can mix and match, in a single project, both high-level concepts and intimate control over every single thing that is happening in your system.
Analytical Modeling of End-to-End Delay in OpenFlow Based NetworksAzeem Iqbal
OpenFlow enabled networks split and separate the data and control planes of traditional networks. This design commodifies network switches and enables centralized control of the network. Control decisions are made by an OpenFlow controller, and locally cached by switches, as directed by controllers. Since controllers are not necessarily co-located with switches that can significantly impact the forwarding delay incurred by packets in switches. Only very few studies have been conducted to evaluate the performance of OpenFlow in terms of end-to-end delay. In this work we develop a stochastic model for the end to end delay in OpenFlow switches based on measurements made in Internetscale experiments performed on three different platforms, i.e. Mininet, the GENI testbed and the OF@TEIN testbed.
Ultra sonic sensor network communicating using NRF 24L01 radioAshok Raj
• Designed mixed signal circuit for ultrasonic sensor network using Arduino shield as programmed processor and NRF24L01 as the radio, PAMAS protocol was used for better efficiency.
• Software used: Arduino
A WSN primary outline issue for a sensor system is protection of the vitality accessible at every sensor node. We propose to convey different, versatile base stations to delay the lifetime of the sensor system. We split the lifetime of the sensor system into equivalent stretches of time known as rounds. Base stations are migrated toward the begin of a round. Our strategy utilizes a whole number straight program to focus new areas for the base stations and in view of steering convention to guarantee vitality proficient directing amid every round. We propose four assessment measurements and look at our answer utilizing these measurements. Taking into account the reproduction results we demonstrate that utilizing various, versatile base stations as per the arrangement given by our plans would altogether expand the lifetime of the sensor system.
Similar to Multipath Load Balancing for SDN Data Plane (20)
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf91mobiles
91mobiles recently conducted a Smart TV Buyer Insights Survey in which we asked over 3,000 respondents about the TV they own, aspects they look at on a new TV, and their TV buying preferences.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
UiPath Test Automation using UiPath Test Suite series, part 4DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap.
The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies.
Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques
What will you get from this session?
1. Insights into SAP testing best practices
2. Heatmap utilization for testing
3. Optimization of testing processes
4. Demo
Topics covered:
Execution from the test manager
Orchestrator execution result
Defect reporting
SAP heatmap example with demo
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Connector Corner: Automate dynamic content and events by pushing a buttonDianaGray10
Here is something new! In our next Connector Corner webinar, we will demonstrate how you can use a single workflow to:
Create a campaign using Mailchimp with merge tags/fields
Send an interactive Slack channel message (using buttons)
Have the message received by managers and peers along with a test email for review
But there’s more:
In a second workflow supporting the same use case, you’ll see:
Your campaign sent to target colleagues for approval
If the “Approve” button is clicked, a Jira/Zendesk ticket is created for the marketing design team
But—if the “Reject” button is pushed, colleagues will be alerted via Slack message
Join us to learn more about this new, human-in-the-loop capability, brought to you by Integration Service connectors.
And...
Speakers:
Akshay Agnihotri, Product Manager
Charlie Greenberg, Host
Generating a custom Ruby SDK for your web service or Rails API using Smithyg2nightmarescribd
Have you ever wanted a Ruby client API to communicate with your web service? Smithy is a protocol-agnostic language for defining services and SDKs. Smithy Ruby is an implementation of Smithy that generates a Ruby SDK using a Smithy model. In this talk, we will explore Smithy and Smithy Ruby to learn how to generate custom feature-rich SDKs that can communicate with any web service, such as a Rails JSON API.
Essentials of Automations: Optimizing FME Workflows with ParametersSafe Software
Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place.
Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects.
Here’s what you’ll gain:
- Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows.
- Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy.
- Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency.
- Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity.
We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic.
Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
Securing your Kubernetes cluster_ a step-by-step guide to success !KatiaHIMEUR1
Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster.
However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks.
In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualityInflectra
In this insightful webinar, Inflectra explores how artificial intelligence (AI) is transforming software development and testing. Discover how AI-powered tools are revolutionizing every stage of the software development lifecycle (SDLC), from design and prototyping to testing, deployment, and monitoring.
Learn about:
• The Future of Testing: How AI is shifting testing towards verification, analysis, and higher-level skills, while reducing repetitive tasks.
• Test Automation: How AI-powered test case generation, optimization, and self-healing tests are making testing more efficient and effective.
• Visual Testing: Explore the emerging capabilities of AI in visual testing and how it's set to revolutionize UI verification.
• Inflectra's AI Solutions: See demonstrations of Inflectra's cutting-edge AI tools like the ChatGPT plugin and Azure Open AI platform, designed to streamline your testing process.
Whether you're a developer, tester, or QA professional, this webinar will give you valuable insights into how AI is shaping the future of software delivery.
Elevating Tactical DDD Patterns Through Object CalisthenicsDorra BARTAGUIZ
After immersing yourself in the blue book and its red counterpart, attending DDD-focused conferences, and applying tactical patterns, you're left with a crucial question: How do I ensure my design is effective? Tactical patterns within Domain-Driven Design (DDD) serve as guiding principles for creating clear and manageable domain models. However, achieving success with these patterns requires additional guidance. Interestingly, we've observed that a set of constraints initially designed for training purposes remarkably aligns with effective pattern implementation, offering a more ‘mechanical’ approach. Let's explore together how Object Calisthenics can elevate the design of your tactical DDD patterns, offering concrete help for those venturing into DDD for the first time!
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
Keynote at DIGIT West Expo, Glasgow on 29 May 2024.
Cheryl Hung, ochery.com
Sr Director, Infrastructure Ecosystem, Arm.
The key trends across hardware, cloud and open-source; exploring how these areas are likely to mature and develop over the short and long-term, and then considering how organisations can position themselves to adapt and thrive.
Key Trends Shaping the Future of Infrastructure.pdf
Multipath Load Balancing for SDN Data Plane
1. Multipath Load Balancing for SDN
Data Plane
For ICONIC 2018, 6-7 Dec 2018, Mon Tresor, Mauritius
Authors: Mpho Nkosi*, Albert A. Lysko, Sabelo Dlamini
CSIR Meraka Institute, Pretoria, South Africa
Email: mnkosi2@csir.co.za
2. 2
Contents
• Introduction
• Focus of the work
• System Setup and Methodology
• Multi-path load balancing algorithm
• Results
• Conclusion
3. 3
Introduction (1 of 2)
• SDN is a key enabling networking paradigm for 5G communications
• Core function of SDN is to separate the control from the data plane based
on abstract representation of network
• Abstraction is achieved through the use of SDN controller which defines the
behaviour of the forwarding plane and routing policies
• For each path, the controller defines a flow together with flow routing
policy. Thus, for each path, only one flow is defined
4. 4
Introduction (2 of 2)
• Load balancing is the method of managing incoming traffic by distributing
and sharing the load fairly among available resources
• In SDN, load balancers are program codes which can easily be
implemented on the SDN controller to efficiently manage network load
• Most SDN controllers come with built in static network load balancers
• Load balancing in SDN controllers is implemented using two approaches: the
stateless or the state-full load balancing
5. 5
Related Work
• Khan et. al, studied dynamic load balancing based on traffic volume by
monitoring link usage and load balancing traffic among available links to
avoid link over loading [1].
• Gupta et. al studied flow statistics based load balancing using POX controller
[2]. In their method, load balancing is performed to avoid server overloading
by fairly sharing server connection requests among multiple servers
6. 6
Focus of this work
• This work studies multi-path load balancing mechanism to reduce network
response time and maximize the overall network performance
• To measure the performance of multipath-based load balancing in a single
centralized OpenDayLight controller network
• The main contribution of this work is a load balancing mechanism for SDN
centralized controller environments which can be employed at any point in
time in a network
7. 7
System setup (1 of 2)
• Two different scenarios with different network topologies were considered:
Scenario 1: a single centralised SDN controller connected to 7 OpenFlow
switches and 8 hosts
Scenario 2: a single centralised SDN controller connected 40 OpenFlow
switches and 81hosts
• Nitrogen version of the open source Opendaylight controller was used in
both scenarios
• Karaf dlux features were used to monitor the topologies, nodes and
controller-switch communications
8. 8
System setup ( 2 of 2)
• PC with Linux Ubuntu 18.04 with 8GB RAM and 2.7GHz processing speed
was used to implement this study
• Mininet SDN emulation tool was used to emulate the network topologies
• Iperf was used to create TCP data streams and to measure the throughput of
the data flow before load balancing and after load balancing.
• Wireshark was also used to analyse packet routing before and after load
balancing
9. 9
Implementation Methodology ( 1 of 2)
• Emulate a network topology using mininet and run mininet ping all to ensure
that nodes and links are up and running.
• Identify a source-destination pair
• Verify that the path is indeed used for transmission for the source-destination
pair using Wireshark
• Create a ping on the source-destination pair to generate traffic and flow
congestion
• Perform an iperf and another ping to measure the latency and bandwidth
utilization on the over loaded flow
10. 10
Implementation Methodology ( 2 of 2)
• Perform load balancing on the congested source-destination pair)
• Perform an iperf and ping again on the source-destination pair to measure
the performance of the load balancing.
11. 11
The multi-path load balancing algorithm
• Takes source-destination pair as input
• Extracts network topology using JSON and REST APIs and perform link,
port MAC and IP mappings together with switch and port connections.
• Extracts ports transmissions rates and stats to understand load on each
port for each flow
• Possible best alternative paths are defined based on lowest flow cost
• Flow cost is calculated as the sum of number of transmitted and received
packets at that time
• Traffic for the source-destination pair is pushed down on to the alternative
flows until all alternative Flows have equal flow cost
14. 14
Results (1 of 4)
0
1
2
3
4
5
6
7
8
9
10
MIN AVG MAX MDEV
Time in ms
average ping for 10packets of size 10240byte
Scenario1: Ping results
Before Load Balancnig
After load balancing
15. 15
Results (2 of 4)
0
1
2
3
4
5
6
7
Transfer(GBytes) Bandwidth(Mbits/sec)
GBvs.MBits/sec
averageiperf for timeinterval of 15sec
Scenario 1: Iperf results
Before Load balanacing
After load balalncing
16. 16
Results (3 of 4)
0
1
2
3
4
5
6
7
8
9
10
MIN AVG MAX MDEV
Timeinms
Averageping for 10 packets of size10240 Byte
Scenario 2:Pingresults
Before Load balancing
After Load Balancing
17. 17
Results (4 of 4)
0
1
2
3
4
5
6
7
Transfer(GBytes) Bandwidth(Mbits/sec)
GBvs.MB/sec
averageiperf for timeinterval of 15sec
Scenario 2: Iperf results
Before Load balanacing
After load balalncing
18. 18
Conclusion
• From scenario 1 and 2, it can be concluded that the load balancer is flexible
for both smaller networks and larger networks. It also can improve network
performance and avoid overall network delay
• However, it was found that for better network improvement, the data plane
should have multiple alternative links so that multiple path paths can be
defined for a routing path
19. 19
References
[1] S. Bhandarkar and K. A. Khan, “Load Balancing in Software-defined Network
( SDN ) Based on Traffic Volume,” vol. 2, no. 7, pp. 72–76, 2015
[2] K. Kaur, S. Kaur, and V. Gupta, “Flow statistics based load balancing in
OpenFlow,” 2016 Int. Conf. Adv. Comput. Commun. Informatics, pp. 378–381,
2016