• Find out about the sFlow instrumentation built into commodity data center network and server infrastructure.
• Understand how sFlow fits into the broader ecosystem of NetFlow, IPFIX, SNMP and DevOps monitoring technologies.
• Case studies demonstrate how sFlow telemetry combined with automation can lower costs, increase performance, and improve security of cloud infrastructure and applications.
The InfluxDB 2.0 Storage Engine | Jacob Marble | InfluxDataInfluxData
The InfluxDB storage engine was completely overhauled for 2.0. Jacob will walk through why we made these changes and discuss architectural considerations in using the new TSM engine.
The agenda of this talk was to introduce MySQL Replication and then follow it up with Multi-threaded slaves(MTS) support. The presentation introduces Multi threading slaves by database which is a part of MySQL-5.6 as well as multi-threading policy introduced in MySQL-5.7.2. Finally there is a brief coverage of the new replication monitoring tables to monitor MySQL Replication. These tables are part of MySQL Performance Schema.
The Next Generation Firewall for Red Hat Enterprise Linux 7 RCThomas Graf
The Linux packet filtering technology, iptables, has its roots in times when networking was relatively simple and network bandwidth was measured in mere megabits. Emerging technologies, such as distributed NAT, overlay networks and containers require enhanced functionality and additional flexibility. In parallel, the next generation of network cards with speeds of 40Gb and 100Gb will put additional pressure on performance.
In the upcoming Red Hat Enterprise Linux 7, a new dynamic firewall service, FirewallD, is planned to provide greater flexibility over iptables by eliminating service disruptions during rule updates, abstraction, and support for different network trust zones. Additionally, a new virtual machine-based packet filtering technology, nftables, addresses the functionality and flexibility requirements of modern network workloads.
In this session you’ll:
Deep dive into the newly introduced packet filtering capabilities of Red Hat Enterprise Linux 7 beta.
Learn best practices.
See the new set of configuration utilities that allow new optimization possibilities.
Choosing an HDFS data storage format- Avro vs. Parquet and more - StampedeCon...StampedeCon
At the StampedeCon 2015 Big Data Conference: Picking your distribution and platform is just the first decision of many you need to make in order to create a successful data ecosystem. In addition to things like replication factor and node configuration, the choice of file format can have a profound impact on cluster performance. Each of the data formats have different strengths and weaknesses, depending on how you want to store and retrieve your data. For instance, we have observed performance differences on the order of 25x between Parquet and Plain Text files for certain workloads. However, it isn’t the case that one is always better than the others.
The InfluxDB 2.0 Storage Engine | Jacob Marble | InfluxDataInfluxData
The InfluxDB storage engine was completely overhauled for 2.0. Jacob will walk through why we made these changes and discuss architectural considerations in using the new TSM engine.
The agenda of this talk was to introduce MySQL Replication and then follow it up with Multi-threaded slaves(MTS) support. The presentation introduces Multi threading slaves by database which is a part of MySQL-5.6 as well as multi-threading policy introduced in MySQL-5.7.2. Finally there is a brief coverage of the new replication monitoring tables to monitor MySQL Replication. These tables are part of MySQL Performance Schema.
The Next Generation Firewall for Red Hat Enterprise Linux 7 RCThomas Graf
The Linux packet filtering technology, iptables, has its roots in times when networking was relatively simple and network bandwidth was measured in mere megabits. Emerging technologies, such as distributed NAT, overlay networks and containers require enhanced functionality and additional flexibility. In parallel, the next generation of network cards with speeds of 40Gb and 100Gb will put additional pressure on performance.
In the upcoming Red Hat Enterprise Linux 7, a new dynamic firewall service, FirewallD, is planned to provide greater flexibility over iptables by eliminating service disruptions during rule updates, abstraction, and support for different network trust zones. Additionally, a new virtual machine-based packet filtering technology, nftables, addresses the functionality and flexibility requirements of modern network workloads.
In this session you’ll:
Deep dive into the newly introduced packet filtering capabilities of Red Hat Enterprise Linux 7 beta.
Learn best practices.
See the new set of configuration utilities that allow new optimization possibilities.
Choosing an HDFS data storage format- Avro vs. Parquet and more - StampedeCon...StampedeCon
At the StampedeCon 2015 Big Data Conference: Picking your distribution and platform is just the first decision of many you need to make in order to create a successful data ecosystem. In addition to things like replication factor and node configuration, the choice of file format can have a profound impact on cluster performance. Each of the data formats have different strengths and weaknesses, depending on how you want to store and retrieve your data. For instance, we have observed performance differences on the order of 25x between Parquet and Plain Text files for certain workloads. However, it isn’t the case that one is always better than the others.
How to build a streaming Lakehouse with Flink, Kafka, and HudiFlink Forward
Flink Forward San Francisco 2022.
With a real-time processing engine like Flink and a transactional storage layer like Hudi, it has never been easier to build end-to-end low-latency data platforms connecting sources like Kafka to data lake storage. Come learn how to blend Lakehouse architectural patterns with real-time processing pipelines with Flink and Hudi. We will dive deep on how Flink can leverage the newest features of Hudi like multi-modal indexing that dramatically improves query and write performance, data skipping that reduces the query latency by 10x for large datasets, and many more innovations unique to Flink and Hudi.
by
Ethan Guo & Kyle Weller
Introducing Multi Valued Vectors Fields in Apache LuceneSease
Since the introduction of native vector-based search in Apache Lucene happened, many features have been developed, but the support for multiple vectors in a dedicated KNN vector field remained to explore. Having the possibility of indexing (and searching) multiple values per field unlocks the possibility of working with long textual documents, splitting them in paragraphs and encoding each paragraph as a separate vector: scenario that is often encountered by many businesses. This talk explores the challenges, the technical design and the implementation activities happened during the work for this contribution to the Apache Lucene project. The audience is expected to get an understanding of how multi-valued fields can work in a vector-based search use-case and how this feature has been implemented.
SIP and DNS - federation, failover, load balancing and moreOlle E Johansson
SIP use DNS to find a server for a specific URI, like sip:alice@example.com. With DNS a SIP service can provide failover, load balancing and much more. SIP without DNS is a broken solution. SIP and DNS rocks!
In this presentation, Kaz Ohta, Kiyoto Tamura, and Ankush Rustagi from Treasure Data describe the company's Cloud Data Warehouse service.
"The Treasure Data Cloud Data Warehouse service enables companies to get big data analytics running in days not months without specialist IT resources and for a tenth the cost of other alternatives. Traditional data warehousing solutions - even modern alternatives such as Hadoop - are too expensive, complex and take too long for many companies to implement, so the idea of quickly launching a data warehouse service that uses the power and economics of the Cloud for companies of any size, opens up a huge potential market."
Learn more at: http://treasure-data.com * Watch the presentation video: http://inside-bigdata.com/?p=3531
How to build a streaming Lakehouse with Flink, Kafka, and HudiFlink Forward
Flink Forward San Francisco 2022.
With a real-time processing engine like Flink and a transactional storage layer like Hudi, it has never been easier to build end-to-end low-latency data platforms connecting sources like Kafka to data lake storage. Come learn how to blend Lakehouse architectural patterns with real-time processing pipelines with Flink and Hudi. We will dive deep on how Flink can leverage the newest features of Hudi like multi-modal indexing that dramatically improves query and write performance, data skipping that reduces the query latency by 10x for large datasets, and many more innovations unique to Flink and Hudi.
by
Ethan Guo & Kyle Weller
Introducing Multi Valued Vectors Fields in Apache LuceneSease
Since the introduction of native vector-based search in Apache Lucene happened, many features have been developed, but the support for multiple vectors in a dedicated KNN vector field remained to explore. Having the possibility of indexing (and searching) multiple values per field unlocks the possibility of working with long textual documents, splitting them in paragraphs and encoding each paragraph as a separate vector: scenario that is often encountered by many businesses. This talk explores the challenges, the technical design and the implementation activities happened during the work for this contribution to the Apache Lucene project. The audience is expected to get an understanding of how multi-valued fields can work in a vector-based search use-case and how this feature has been implemented.
SIP and DNS - federation, failover, load balancing and moreOlle E Johansson
SIP use DNS to find a server for a specific URI, like sip:alice@example.com. With DNS a SIP service can provide failover, load balancing and much more. SIP without DNS is a broken solution. SIP and DNS rocks!
In this presentation, Kaz Ohta, Kiyoto Tamura, and Ankush Rustagi from Treasure Data describe the company's Cloud Data Warehouse service.
"The Treasure Data Cloud Data Warehouse service enables companies to get big data analytics running in days not months without specialist IT resources and for a tenth the cost of other alternatives. Traditional data warehousing solutions - even modern alternatives such as Hadoop - are too expensive, complex and take too long for many companies to implement, so the idea of quickly launching a data warehouse service that uses the power and economics of the Cloud for companies of any size, opens up a huge potential market."
Learn more at: http://treasure-data.com * Watch the presentation video: http://inside-bigdata.com/?p=3531
Monitor OpenStack Environments from the bottom up and front to backIcinga
Talk given by Thomas Stocking at Icinga Camp San Francisco 2016 - https://www.icinga.org/community/events/archive/2016-archive/icinga-camp-san-francisco/
Data collection and storage is a primary challenge for any big data architecture. In this session, we will describe the different types of data that customers are handling to drive high-scale workloads on AWS, and help you choose the best approach for your workload. We will cover optimization techniques that improve performance and reduce the cost of data ingestion.AWS services to be covered include: Amazon S3, DynamoDB, and Kinesis.
Cloud Network Virtualization with Juniper Contrailbuildacloud
Description: Contrail Technology will be discussed covering architecture, capabilities and use cases. It will be followed by a demonstration on current Contrail implementation on CloudStack/Openstack.
Parantap works as a Sr. Director of Solutions Engineering for Contrail Product within Juniper. Before Juniper, Parantap led the network architecture team for Microsoft Online Services (Windows Azure, MS Bing). Prior to Microsoft, Parantap worked as a core engineering manager for UUNet Technologies building Internet backbones.
Sergei Gotchev, Juniper Networks
Juniper Day, Praha, 13.5.2015
Jestliže SlideShare nezobrazí prezentaci korektně, můžete si ji stáhnout ve formátu .ppsx nebo .pdf (kliknutím na tlačitko v dolní liště snímků).
Architecting and Tuning IIB/eXtreme Scale for Maximum Performance and Reliabi...Prolifics
Abstract: Recent projects have stressed the "need for speed" while handling large amounts of data, with near zero downtime. An analysis of multiple environments has identified optimizations and architectures that improve both performance and reliability. The session covers data gathering and analysis, discussing everything from the network (multiple NICs, nearby catalogs, high speed Ethernet), to the latest features of extreme scale. Performance analysis helps pinpoint where time is spent (bottlenecks) and we discuss optimization techniques (MQ tuning, IIB performance best practices) as well as helpful IBM support pacs. Log Analysis pinpoints system stress points (e.g. CPU starvation) and steps on the path to near zero downtime.
[DSC Europe 23] Pramod Immaneni - Real-time analytics at IoT scaleDataScienceConferenc1
Rivian makes adventurous electric vehicles with a mission of a sustainable planet and keeping the world adventurous forever. Rivian's vehicles are born in the cloud and embody tenets of a software defined vehicle, where not only the user accessible features such as infotainment are software driven and updated, but also internals aspects such as vehicle dynamics. Real-time instrumentation and telemetry are the key underpinnings that make all this possible. Rivian has built a cutting-edge Real-time stack using a combination of open-source technologies like Kafka, Flink and Druid and in house services. This talk will go into how these are combined and leveraged to deliver real-time analytics.
OpManager is integrated network management software that offers network monitoring, server monitoring, bandwidth analysis, configuration management, firewall log analysis, server management and IP & switch port management.
Hpe service virtualization 3.8 what's new chicago admJeffrey Nunn
Service Virtualization is an HPE branded solution that helps simulate and emulate the behavior of specific components in heterogeneous component-based applications such as API-driven apps, ERP apps, cloud-based apps, and web services/service-oriented architectures (SOA).
Value Proposition
Empowers developers and testers to easily automate, predict, accelerate and scale their application testing and delivery through virtualization and simulation of dependent components and services that are either off limits, unavailable, inaccessible, or with costly fees to access.
OpManager is an integrated network management tool that helps you monitor your network, physical & virtual servers, bandwidth, configurations, firewall, switch ports and IP addresses
Enhanced Enterprise Intelligence with your personal AI Data Copilot.pdfGetInData
Recently we have observed the rise of open-source Large Language Models (LLMs) that are community-driven or developed by the AI market leaders, such as Meta (Llama3), Databricks (DBRX) and Snowflake (Arctic). On the other hand, there is a growth in interest in specialized, carefully fine-tuned yet relatively small models that can efficiently assist programmers in day-to-day tasks. Finally, Retrieval-Augmented Generation (RAG) architectures have gained a lot of traction as the preferred approach for LLMs context and prompt augmentation for building conversational SQL data copilots, code copilots and chatbots.
In this presentation, we will show how we built upon these three concepts a robust Data Copilot that can help to democratize access to company data assets and boost performance of everyone working with data platforms.
Why do we need yet another (open-source ) Copilot?
How can we build one?
Architecture and evaluation
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Data and AI
Round table discussion of vector databases, unstructured data, ai, big data, real-time, robots and Milvus.
A lively discussion with NJ Gen AI Meetup Lead, Prasad and Procure.FYI's Co-Found
Unleashing the Power of Data_ Choosing a Trusted Analytics Platform.pdfEnterprise Wired
In this guide, we'll explore the key considerations and features to look for when choosing a Trusted analytics platform that meets your organization's needs and delivers actionable intelligence you can trust.
Learn SQL from basic queries to Advance queriesmanishkhaire30
Dive into the world of data analysis with our comprehensive guide on mastering SQL! This presentation offers a practical approach to learning SQL, focusing on real-world applications and hands-on practice. Whether you're a beginner or looking to sharpen your skills, this guide provides the tools you need to extract, analyze, and interpret data effectively.
Key Highlights:
Foundations of SQL: Understand the basics of SQL, including data retrieval, filtering, and aggregation.
Advanced Queries: Learn to craft complex queries to uncover deep insights from your data.
Data Trends and Patterns: Discover how to identify and interpret trends and patterns in your datasets.
Practical Examples: Follow step-by-step examples to apply SQL techniques in real-world scenarios.
Actionable Insights: Gain the skills to derive actionable insights that drive informed decision-making.
Join us on this journey to enhance your data analysis capabilities and unlock the full potential of SQL. Perfect for data enthusiasts, analysts, and anyone eager to harness the power of data!
#DataAnalysis #SQL #LearningSQL #DataInsights #DataScience #Analytics
Adjusting OpenMP PageRank : SHORT REPORT / NOTESSubhajit Sahu
For massive graphs that fit in RAM, but not in GPU memory, it is possible to take
advantage of a shared memory system with multiple CPUs, each with multiple cores, to
accelerate pagerank computation. If the NUMA architecture of the system is properly taken
into account with good vertex partitioning, the speedup can be significant. To take steps in
this direction, experiments are conducted to implement pagerank in OpenMP using two
different approaches, uniform and hybrid. The uniform approach runs all primitives required
for pagerank in OpenMP mode (with multiple threads). On the other hand, the hybrid
approach runs certain primitives in sequential mode (i.e., sumAt, multiply).
Techniques to optimize the pagerank algorithm usually fall in two categories. One is to try reducing the work per iteration, and the other is to try reducing the number of iterations. These goals are often at odds with one another. Skipping computation on vertices which have already converged has the potential to save iteration time. Skipping in-identical vertices, with the same in-links, helps reduce duplicate computations and thus could help reduce iteration time. Road networks often have chains which can be short-circuited before pagerank computation to improve performance. Final ranks of chain nodes can be easily calculated. This could reduce both the iteration time, and the number of iterations. If a graph has no dangling nodes, pagerank of each strongly connected component can be computed in topological order. This could help reduce the iteration time, no. of iterations, and also enable multi-iteration concurrency in pagerank computation. The combination of all of the above methods is the STICD algorithm. [sticd] For dynamic graphs, unchanged components whose ranks are unaffected can be skipped altogether.
Influence of Marketing Strategy and Market Competition on Business Plan
Network visibility and control using industry standard sFlow telemetry
1. Network visibility and control using
industry standard sFlow telemetry
Peter Phaal
InMon Corp.
March, 2016
Twitter: @sFlow
Blog: blog.sflow.com
San Francisco Network Visibility Meetup
6. Controllability and Observability
Basic concept is simple, a stable feedback control system requires:
1. ability to influence all important system states (controllable)
2. ability to monitor all important system states (observable)
7. It’s hard to stay on the road if you can’t see the
road, or keep to the speed limit without a
speedometer
It’s hard to stay on the road or maintain
speed if your brakes, engine or steering fail
Controllability and Observability driving example
Observability
Controllability
States location, speed, direction, ...
Tule fog in California Central Valley
8. Effect of delay on stability
Measurement delay Planning delay
Time
Configuration delayDisturbance Response delay
EffectLoop delay
DDoS launched Identify target, attacker Black hole, mark, re-route? Switch CLI commands Route propagation Traffic dropped
Components of loop delay
e.g. Slow reaction time causes
tired / drunk / distracted
driver to weave, very slow
reaction time and they leave
the road
9. What is sFlow?
“In God we trust. All others bring data.”
Dr. Edwards Deming
11. Open source agents for hosts, hypervisors and applications
Host sFlow project (http://sflow.net) is center of an ecosystem
of related open source projects embedding sFlow in popular
operating systems and applications
Host agent extends network visibility into public / private cloud
13. Simple
- standard structures - densely packed blocks of counters
- extensible (tag, length, value)
- RFC 1832: XDR encoded (big endian, quad-aligned, binary) - simple to encode /
decode
- unicast UDP transport
Minimal configuration
- collector address
- polling interval
Cloud friendly
- flat, two tier architecture: many embedded agents → central “smart” collector
- sFlow agents automatically start sending metrics on startup, automatically discovered
- eliminates complexity of maintaining polling daemons (and associated configurations)
Scaleable push protocol
14. • Counters tell you there is a
problem, but not why.
• Counters summarize
performance by dropping high
cardinality attributes:
- IP addresses
- URLs
- Memcache keys
• Need to be able to efficiently
disaggregate counter by
attributes in order to
understand root cause of
performance problems.
• How do you get this data when
there are millions of
transactions per second?
Counters aren’t enough
Why the spike in traffic?
(100Gbit link carrying 14,000,000 packets/second)
15. • Random sampling is lightweight
• Critical path roughly cost of
maintaining one counter:
if(--skip == 0) sample();
• Sampling is easy to distribute
among modules, threads,
processes without any
synchronization
• Minimal resources required to
capture attributes of sampled
transactions
• Easily identify top keys,
connections, clients, servers, URLs
etc.
• Unbiased results with known
accuracy
Break out traffic by client, server and port
(graph based on samples from100Gbit link carrying 14,000,000 packets/second)
sFlow also exports random samples
16. Integrated data model
Packet Header
Source Destination
TCP/UDP Socket TCP/UDP Socket
MAC Address MAC Address
Sampled Packet Headers
+
Forwarding State
I/F Counters
NETWORK
HOST
CPU
Memory
I/O
Adapter MACs
APPLICATION
Sampled Transactions
Transaction Counters
TCP/UDP Socket
Independent agents sFlow analyzer joins data for integrated view
18. Picking the right tools
“This is the Unix philosophy: Write programs that do one
thing and do it well. Write programs to work together.”
Doug McIlroy
19. packets
decode hash sendflow cache flushsample
Flow
Records
flow cache embedded on switchswitch
NetFlow
IPFIX
…
decode hash sendflow cache flush
Flow
Records
packets
send
polli/f counters
sample
multiple switches export sFlow
packets
send
polli/f counters
sample
...
centralized software flow cache
switch
switch
JSON/REST
NetFlow
IPFIX
…
• Reduce ASIC cost / complexity
• Fast response (data not sitting on switch)
• Centralized, network-wide visibility
• Increase flexibility → software defined analytics
Move flow cache from ASIC to external software
Scale-out alternative to SNMP polling
Traffic analytics with sFlow
20. sFlow-RT.com analytics engine
• Low latency flow analytics for real-time control applications
• Disaggregates flow cache from database. Choose external
database(s) for history (InfluxDB, Logstash, etc.)
• Programmable analytics pipeline through open APIs
21. RESTful API for defining flows
http://blog.sflow.com/2013/08/restflow.html
curl -H "Content-Type:application/json" -X PUT —data
'{"keys":"ipsource,ipdestination,tcpsourceport,tcpdestinationport",
"value":"bytes", "ipfixCollectors":["10.0.0.1"]}'
http://127.0.0.1:8008/flow/tcp/json
curl -H "Content-Type:application/json" -X PUT --data
'{"keys":"ipdestination,icmpunreachableport", "value":"frames"}'
http://127.0.0.1:8008/flow/unreachableport/json
• Instantly enables network wide monitoring of flows
• All switches, all ports, including hosts and virtual switches
• Contrast with task of re-configuring Flexible Netflow/IPFIX caches on
every switch in multi-vendor network. How many simultaneous flow
definitions are allowed? What key / value combinations are allowed?
curl -H "Content-Type:application/json" -X PUT --data
'{"value":"frames"}'
http://127.0.0.1:8008/flow/frames/json
22. InMon sFlow-RT
active timeout active timeout
NetFlow
Open
vSwitch
SolarWinds Real-Time NetFlow Analyzer
• sFlow does not use flow cache, so realtime charts more accurately reflect traffic trend
• NetFlow spikes caused by flow cache active-timeout for long running connections
Rapid detection of large flows
Flow cache active timeout delays large flow detection,
limits value of signal for real-time control applications
23. Counters and packet samples
http://blog.sflow.com/2013/02/measurement-delay-counters-vs-packet.html
• Packet samples give a fast signal that operates at scale
• Counters are maintained in hardware and provide precise traffic totals.
• Counters capture rare events, like packet discards, that can severely
impact performance.
• Counters report important link state information, like link speed, LAG
group membership etc.
25. Data models and transports
sFlow SNMP
NetFlow
version 5
OpenConfig
Telemetry IPFIX syslog
Model
standard
measurements
published by
sFlow.org,
Dataplane
focus: based on
IEEE, IETF,
APIs (MIB-2,
LAG-MIB,
libvirt, JMX, …)
standard
MIBs
defined
by IETF
standard
tcp / udp /
icmp flow
record
defined by
Cisco
Telemetry defined
as part of YANG
configuration
models by
OpenConfig.org
Control plane
focus: BGP,
MPLS, VLAN, etc.
Encoding
XDR
(RFC 4506)
ASN1
(IETF)
NetFlow
(Cisco)
protobufs,
JSON,
NetConf
IPFIX
(IETF)
Syslog
(RFC 5424)
Transport UDP UDP UDP UDP, HTTP
SCTP,
TCP,
UDP
UDP
Mode Push Pull Push Push Push Push
Easy to combine multiple data sources if you disaggregate tool chain
e.g. separate agents from collectors, feed data from all sources into InfluxDB / Logstash
27. Network visibility for DevOps tools
• Streaming filtering and summarization reduces data volume
and increases scaleability of backend tools
• Streaming flow analytics to generate application metrics
28. Feedback control of cloud infrastructure
“You can’t control what you can’t measure”
Tom DeMarco
29. Cloud depends on network
• Server costs (both capex and power consumption) far exceed networking costs in the data center.
• Network congestion caused server to wait, resulting in poor utilization of cloud infrastructure.
• Optimize network to increase data center efficiency
http://perspectives.mvdirona.com/2010/09/overall-data-center-costs/
“Typically the resource that is most scarce is the network.”
Amin Vahdat, Google, ONS2015 Keynote
http://blog.sflow.com/2015/06/optimizing-software-defined-data-center.html
35. ECMP monitoring challenge
• large number of links, 12
x 10G links
• all links need to be
monitored continuously,
180G total bandwidth
• real-time detection of
congested links
• real-time detection of
Elephant flows
http://blog.sflow.com/2015/03/ecmp-visibility-with-cumulus-linux.html
36. Fabric level performance metrics
• Fabric View application runs on
sFlow-RT
• Downloadable from sFlow-RT.com,
includes captured data set from 4
node 10G ECMP fabric
• Elephant collisions on spine links
occur frequently
• Collisions halve throughput
• Collisions cause packet discards
http://blog.sflow.com/2015/10/fabric-view.html
47. Open Networking Summit SDN Idol winning solution
Real-time SDN Analytics for DDoS mitigation
48. DDoS Mitigation Market Opportunity
DDoS Attack Megatrends [Reference 1]
• High bandwidth, volumetric infrastructure layer (Layer 3 & 4) attacks increased
approximately 30 percent
• DDoS attack volume also increased month-to-month in 2013, with 10 out of
12 months showing higher attack volume compared to 2012
• Average DDoS attack sizes continued to increase – many over 100 Gbps, the
largest peaking at 179 Gbps
DDoS Mitigation Market Growth
• $870M market by 2017, 18.2% CAGR – Source: IDC:Worldwide DDoS Prevention
Products and Services 2013-2017 Forecast
• $1049M market by 2017, 25% CAGR – Source: Infonetics: Global DDoS Prevention
Appliances 2012-2017 Forecast
Reference 1: Top DDoS Attack Trends http://www.itbriefcase.net/top-ddos-attack-trends-for-2013
49. DDoS Mitigation Use Case (1)
ISP 1
ISP 2
ISP N
• ISP/IX is uniquely positioned to protect customers from DDoS flood attacks
• New revenue from DDoS mitigation service + differentiates ISP/IX service
Attacker
User Prevent attack from
overwhelming customer
access link
Filter attack traffic in
real-time
Customer network
DDoS target host
Attack on single host can take out entire
customer data center. Customer cannot
mitigate flood attack without upstream help
ISP / IX
ISP/IX Market Segment
50. Customer
portal
DDoS Mitigation Service
Web UI + RESTful programmatic API
• real-time TopN analytics
• programmable filtering of traffic
• set thresholds + automatic blocking
Real-time sFlow visibility, Hybrid OpenFlow Control capability of Brocade switches/routers
REST API
InMonsFlow-RT
REST API
OpenFlowController
DDoS Mitigation
Application
Customer
Network
Internet
1. Flood
attack
overloads
customer
port
2. Attack maps to large flows
[Ref. 2]. sFlow-RT detects
attack (maps to large flows)
and characterizes attack
(srcip, dstip, protocol, ports,
etc.)
3. mitigation application takes signature, applies
customer policy, selects optimal control and push
OpenFlow rule(s) to switch(es)
5. OpenFlow rule(s)
applied to switch
forwarding path to drop /
mark traffic and protect link
HTTPS HTTPS
4. Controller pushes
OpenFlow rule(s) to
switch(es)
OpenFlow 1.3 Match Fields
line rate filtering using Brocade switches
Reference 2: IETF I2RS Working Group Draft - https://ietf.org/doc/draft-krishnan-i2rs-large-flow-use-case/
57. Comments
• sFlow instrumentation is widely available in switches
http://sflow.org/products/network.php
• Host sFlow (sFlow.net) agent extends visibility into
servers (works with libpcap, iptables, Open vSwitch to
efficiently sample packets in host data plane)
• Common data model ensures strong interoperability
across sFlow data sources
• Streaming counter and packet telemetry across network,
compute and application tiers makes data center
observable
• Observability makes it possible to apply feedback controls
59. Host sFlow monitoring of Linux datapath
Technology Reference
Adapter, bridge,
macvlan, ipvlan
Berkeley Packet
Filter (BPF) sampling
function
http://blog.sflow.com/
2016/02/linux-bridge-
macvlan-ipvlan-adapters.html
Open vSwitch
Kernel datapath
has sFlow support
http://openvswitch.org/
support/config-
cookbooks/sflow/
Linux Firewall
iptables statistic
module random
function with ulog
http://blog.sflow.com/
2010/12/ulog.html
Top of Rack
Switch
ASIC provides
wirespeed monitoring
of attached servers
http://blog.sflow.com/
2010/04/hybrid-server-
monitoring.html
Efficient monitoring of high traffic production workloads
60. Open vSwitch Fall Conference
New OVS instrumentation features aimed at
real-time monitoring of virtual networks
69. Hybrid OpenFlow ECMP testbed
http://blog.sflow.com/2015/01/hybrid-openflow-ecmp-testbed.html
http://mininet.org/
• Simulated ECMP
network for developing
visibility and control
applications
• sFlow support in Open
vSwitch
• OpenFlow for control
70. The sFlow Standard: Scalable, Unified Monitoring of
Networks, Systems and Applications
2012 Velocity Conference