The document discusses network flow analysis using flow-tools. It provides instructions on capturing and analyzing network flow data from routers and switches using flow-tools, a Linux-based set of tools. Specific steps outlined include configuring devices to export NetFlow data, capturing flows to files using flow-capture, and analyzing the flows using utilities like flow-cat and flow-stat to identify top talkers, protocols, ports, and traffic patterns. Examples demonstrate how flow-tools can be used to detect anomalies like denial of service attacks by analyzing source and destination IP addresses and ports in captured network flow data.
Once in a while network engineers working in IIGs or ISPs in Bangladesh have to face a phenomenon: a switching loop . In our part of the network backbone which is switch based ,we have all the recommended loop prevention mechanisms. Even after that sometimes broadcast storm takes places. The paper discusses my findings on what may have caused this occurrences and my recommendations. I wrote about this topic for the first time 14 months back on LinkedIn as an article. I believe the topic is still relevant.
Once in a while network engineers working in IIGs or ISPs in Bangladesh have to face a phenomenon: a switching loop . In our part of the network backbone which is switch based ,we have all the recommended loop prevention mechanisms. Even after that sometimes broadcast storm takes places. The paper discusses my findings on what may have caused this occurrences and my recommendations. I wrote about this topic for the first time 14 months back on LinkedIn as an article. I believe the topic is still relevant.
Haystack XR Mode radically improves the range of IoT devices using Semtech's LoRa radio. Using the same error correction used in deep space probes along with other techniques, Haystack offers the longest range, lowest power, and lowest latency networking stack for LoRa today.
BGP Traffic Engineering with SDN Controller, by Shaowen Ma.
A presentation given at APRICOT 2016’s Software Defined Networking session on 24 February 2016.
Frank Brockners' presentation from the 2017 Open Networking Summit.
While troubleshooting or planning, did you ever wish to get full insight into which paths *all* your packets take in your network, understand whether your SLA is really in place, or were you ever asked to prove that your traffic really follows the path you specified by service chaining or traffic engineering? We approach this problem by adding meta-data to *all* packets. In-band OAM adds forwarding path information and other information/stats to every data packet - as opposed to relying on probe packets, which is the traditional method that tools like ping or traceroute use. This session will introduce In-band OAM, explain the technology and outline the reference implementation in FD.io/VPP and OpenDaylight using a example demos.
Haystack XR Mode radically improves the range of IoT devices using Semtech's LoRa radio. Using the same error correction used in deep space probes along with other techniques, Haystack offers the longest range, lowest power, and lowest latency networking stack for LoRa today.
BGP Traffic Engineering with SDN Controller, by Shaowen Ma.
A presentation given at APRICOT 2016’s Software Defined Networking session on 24 February 2016.
Frank Brockners' presentation from the 2017 Open Networking Summit.
While troubleshooting or planning, did you ever wish to get full insight into which paths *all* your packets take in your network, understand whether your SLA is really in place, or were you ever asked to prove that your traffic really follows the path you specified by service chaining or traffic engineering? We approach this problem by adding meta-data to *all* packets. In-band OAM adds forwarding path information and other information/stats to every data packet - as opposed to relying on probe packets, which is the traditional method that tools like ping or traceroute use. This session will introduce In-band OAM, explain the technology and outline the reference implementation in FD.io/VPP and OpenDaylight using a example demos.
Applied Detection and Analysis Using Flow Data - MIRCon 2014chrissanders88
In this presentation, Chris Sanders and Jason Smith discuss the importance of using flow data for network security analysis. Flow data is discussed from the viewpoints of collection, detection, and analysis. We also discuss the FlowPlotter tool, and the use of FlowBAT, a graphical flow analysis GUI we've created.
This gives an overall idea about wireshark design and how to capture packets using wireshark, tcpdump and tshark. It also covers basics behind measuring network performance and tools to use such as bmon and iperf.
Application Visibility and Experience through Flexible NetflowCisco DevNet
The world of applications is changing rapidly in the enterprise; from the way applications are increasingly hosted in the cloud, the diverse nature of apps and to the way they are consumed by many devices. The need for organizations and network administrators is to focus on "Fast IT" - "Innovation in the Enterprise" is growing, which means having to spend less time on daily operations, maintenance and troubleshooting and more time on delivering business value with newer services. Cisco AVC with its NBAR2 technology is designed to detect applications and measure application performance through measuring round trip time, retransmission rates, jitter, delay, packet loss, MoS, URL statistics etc. Those details are transmitted using Flexible Netflow/IPFIX, so partners could leverage the data for application usage reporting, performance reporting and troubleshooting application issues to deliver best possible application experience.
Watch the DevNet 2047 replay from the Cisco Live On-Demand Library at: https://www.ciscolive.com/online/connect/sessionDetail.ww?SESSION_ID=92664&backBtn=true
Check out more and register for Cisco DevNet: http://ow.ly/jCNV3030OfS
NetFlow Monitoring for Cyber Threat DefenseCisco Canada
Recent trends have led to the erosion of the security perimeter and increasingly attackers are gaining operational footprints on the network interior. For more information, please visit our website: http://www.cisco.com/web/CA/index.html
Fast Streaming into Clickhouse with Apache PulsarTimothy Spann
https://github.com/tspannhw/SpeakerProfile/tree/main/2022/talks
Fast Streaming into Clickhouse with Apache Pulsar
https://github.com/tspannhw/FLiPC-FastStreamingIntoClickhouseWithApachePulsar
https://www.meetup.com/San-Francisco-Bay-Area-ClickHouse-Meetup/events/285271332/
Fast Streaming into Clickhouse with Apache Pulsar - Meetup 2022
StreamNative - Apache Pulsar - Stream to Altinity Cloud - Clickhouse
May the 4th Be With You!
04-May-2022 Clickhosue Meetup
CREATE TABLE iotjetsonjson_local
(
uuid String,
camera String,
ipaddress String,
networktime String,
top1pct String,
top1 String,
cputemp String,
gputemp String,
gputempf String,
cputempf String,
runtime String,
host String,
filename String,
host_name String,
macaddress String,
te String,
systemtime String,
cpu String,
diskusage String,
memory String,
imageinput String
)
ENGINE = MergeTree()
PARTITION BY uuid
ORDER BY (uuid);
CREATE TABLE iotjetsonjson ON CLUSTER '{cluster}' AS iotjetsonjson_local
ENGINE = Distributed('{cluster}', default, iotjetsonjson_local, rand());
select uuid, top1pct, top1, gputempf, cputempf
from iotjetsonjson
where toFloat32OrZero(top1pct) > 40
order by toFloat32OrZero(top1pct) desc, systemtime desc
select uuid, systemtime, networktime, te, top1pct, top1, cputempf, gputempf, cpu, diskusage, memory,filename
from iotjetsonjson
order by systemtime desc
select top1, max(toFloat32OrZero(top1pct)), max(gputempf), max(cputempf)
from iotjetsonjson
group by top1
select top1, max(toFloat32OrZero(top1pct)) as maxTop1, max(gputempf), max(cputempf)
from iotjetsonjson
group by top1
order by maxTop1
Tim Spann
Developer Advocate
StreamNative
Network Security and Visibility through NetFlowLancope, Inc.
With the rise of disruptive forces such as cloud computing and mobile technology, the enterprise network has become larger and more complex than ever before. Meanwhile, sophisticated cyber-attackers are taking advantage of the expanded attack surface to gain access to internal networks and steal sensitive data.
Perimeter security is no longer enough to keep threat actors out, and organizations need to be able to detect and mitigate threats operating inside the network. NetFlow, a context-rich and common source of network traffic metadata, can be utilized for heightened visibility to identify attackers and accelerate incident response.
Join Richard Laval to discuss the security applications of NetFlow using StealthWatch. This session will cover:
- An overview of NetFlow, what it is, how it works, and how it benefits security
- Design, deployment, and operational best practices for NetFlow security monitoring
- How to best utilize NetFlow and identity services for security telemetry
- How to investigate and identify threats using statistical analysis of NetFlow telemetry
This webinar explains why PISA chips are inevitable, provides overview of machine architecture of such switches, presents a brief primer on the P4 language with sample programs for a variety of networks and demonstrates a powerful network diagnostics application implemented in P4.
Programmability in SDNs is confined to the network control plane. The forwarding plane is still largely dictated by fixed-function switching chips. Our goal is to change that, and to allow programmers to define how packets are to be processed all the way down to the wire.
This is made possible by a new generation of high-performance forwarding chips. At the high-end, PISA (Protocol-Independent Switch Architecture) chips promise multi-Tb/s of packet processing. At the mid- and low-end of the performance spectrum, CPUs, GPUs, FPGAs, and NPUs already offer great flexibility with performance of a few tens to hundreds of Gb/s.
In addition to programmable forwarding chips, we also need a high-level language to dictate the forwarding behavior in a target independent fashion. "P4" (www.p4.org) is such a language. In P4, the programer declares how packets are to be processed, and a compiler generates a configuration for a PISA chip, or a programmable target in general. For example, the programmer might program the switch to be a top-of-rack switch, a firewall, or a load-balancer; and might add features to run automatic diagnostics and novel congestion control algorithms.
This is a tutorial for implementing application level traffic analyzer by using SF-TAP flow abstractor.
http://sf-tap.github.io/
https://github.com/SF-TAP/
https://github.com/SF-TAP/flow-abstractor
https://www.usenix.org/conference/lisa15/conference-program/presentation/takano
http://ytakano.github.io/
APNIC Foundation, presented by Ellisha Heppner at the PNG DNS Forum 2024APNIC
Ellisha Heppner, Grant Management Lead, presented an update on APNIC Foundation to the PNG DNS Forum held from 6 to 10 May, 2024 in Port Moresby, Papua New Guinea.
# Internet Security: Safeguarding Your Digital World
In the contemporary digital age, the internet is a cornerstone of our daily lives. It connects us to vast amounts of information, provides platforms for communication, enables commerce, and offers endless entertainment. However, with these conveniences come significant security challenges. Internet security is essential to protect our digital identities, sensitive data, and overall online experience. This comprehensive guide explores the multifaceted world of internet security, providing insights into its importance, common threats, and effective strategies to safeguard your digital world.
## Understanding Internet Security
Internet security encompasses the measures and protocols used to protect information, devices, and networks from unauthorized access, attacks, and damage. It involves a wide range of practices designed to safeguard data confidentiality, integrity, and availability. Effective internet security is crucial for individuals, businesses, and governments alike, as cyber threats continue to evolve in complexity and scale.
### Key Components of Internet Security
1. **Confidentiality**: Ensuring that information is accessible only to those authorized to access it.
2. **Integrity**: Protecting information from being altered or tampered with by unauthorized parties.
3. **Availability**: Ensuring that authorized users have reliable access to information and resources when needed.
## Common Internet Security Threats
Cyber threats are numerous and constantly evolving. Understanding these threats is the first step in protecting against them. Some of the most common internet security threats include:
### Malware
Malware, or malicious software, is designed to harm, exploit, or otherwise compromise a device, network, or service. Common types of malware include:
- **Viruses**: Programs that attach themselves to legitimate software and replicate, spreading to other programs and files.
- **Worms**: Standalone malware that replicates itself to spread to other computers.
- **Trojan Horses**: Malicious software disguised as legitimate software.
- **Ransomware**: Malware that encrypts a user's files and demands a ransom for the decryption key.
- **Spyware**: Software that secretly monitors and collects user information.
### Phishing
Phishing is a social engineering attack that aims to steal sensitive information such as usernames, passwords, and credit card details. Attackers often masquerade as trusted entities in email or other communication channels, tricking victims into providing their information.
### Man-in-the-Middle (MitM) Attacks
MitM attacks occur when an attacker intercepts and potentially alters communication between two parties without their knowledge. This can lead to the unauthorized acquisition of sensitive information.
### Denial-of-Service (DoS) and Distributed Denial-of-Service (DDoS) Attacks
1.Wireless Communication System_Wireless communication is a broad term that i...JeyaPerumal1
Wireless communication involves the transmission of information over a distance without the help of wires, cables or any other forms of electrical conductors.
Wireless communication is a broad term that incorporates all procedures and forms of connecting and communicating between two or more devices using a wireless signal through wireless communication technologies and devices.
Features of Wireless Communication
The evolution of wireless technology has brought many advancements with its effective features.
The transmitted distance can be anywhere between a few meters (for example, a television's remote control) and thousands of kilometers (for example, radio communication).
Wireless communication can be used for cellular telephony, wireless access to the internet, wireless home networking, and so on.
Multi-cluster Kubernetes Networking- Patterns, Projects and GuidelinesSanjeev Rampal
Talk presented at Kubernetes Community Day, New York, May 2024.
Technical summary of Multi-Cluster Kubernetes Networking architectures with focus on 4 key topics.
1) Key patterns for Multi-cluster architectures
2) Architectural comparison of several OSS/ CNCF projects to address these patterns
3) Evolution trends for the APIs of these projects
4) Some design recommendations & guidelines for adopting/ deploying these solutions.
Bridging the Digital Gap Brad Spiegel Macon, GA Initiative.pptxBrad Spiegel Macon GA
Brad Spiegel Macon GA’s journey exemplifies the profound impact that one individual can have on their community. Through his unwavering dedication to digital inclusion, he’s not only bridging the gap in Macon but also setting an example for others to follow.
This 7-second Brain Wave Ritual Attracts Money To You.!nirahealhty
Discover the power of a simple 7-second brain wave ritual that can attract wealth and abundance into your life. By tapping into specific brain frequencies, this technique helps you manifest financial success effortlessly. Ready to transform your financial future? Try this powerful ritual and start attracting money today!
3. NetworkFlow
A flow is a set of packets with common characteristics within a given time frame and a
given direction.
In packet switching networks, traffic flow, packet flow or network flow is a
sequence of packets from a source computer to a destination, which maybe
another host, a multicast group, or a broadcast domain.
RFC 2722 defines traffic flow as "A TRAFFIC FLOW is an artificial logical
equivalent to a call or connection, belonging to a (user-specified) METERED
TRAFFIC GROUP."
5. NetFlow
NetFlow was introduced in Cisco routers first to get the traffic informationfrom one or
many source/s to one or many destination/s.
Also supported by Juniper, Mikrotiketc.
Jflow or cflowd for Juniper Networks
NetStream for 3Com/HP
NetStream for Huawei Technologies
Cflowd for Alcatel-Lucent
Rflow for Ericsson
AppFlow Citrix
Traffic Flow MikroTik
sFlow vendors include: AlcatelLucent, Cisco, Dell, D-Link, Fortinet, Hewlett-Packard,
Huawei, IBM, Juniper, NEC, Netgear, ZTE etc
6. NetFlow
Version Comment
v1
First implementation, now obsolete, and restricted to IPv4 (without IP mask and
AS Numbers).
v2 Cisco internal version, never released.
v3 Cisco internal version, never released.
v4 Cisco internal version, never released.
v5
Most common version, available on many routers from different brands, but
restricted to IPv4 flows.
v6 No longer supported by Cisco.
v7 Like version 5 with a source router field. Used on Cisco Catalyst switches.
v8
Several aggregation form, but only for information that is already present in
version 5 records
v9
Template Based, available on some recent routers. Mostly used to report flows like
IPv6, MPLS, or even plain IPv4 with BGP nexthop.
v10 Used for identifying IPFIX - IP Flow Information Export.
7. Cisco Configuration
ip flow-export version 5 origin-as
ip flow-export source Loopback0
ip flow-export destination[ServerIP] 3000
interface TenGigabitEthernet1/0/0
ip flow ingress
ip flow egress
8. Juniper Configuration
set firewallfilter test-flow term 1 then sample
set firewallfilter test-flow term 1 then accept
set interfaces ge-0/0/0 unit 0 family inet filter input test-flow
set interfaces ge-0/0/0 unit 0 family inet filter output test-flow
set forwarding-optionssampling input rate 1000
set forwarding-optionssampling familyinet output flow-server [ServerIp] port 3000
set forwarding-optionssampling familyinet output flow-server [ServerIp] version 5
9. Server
First Check if you are receiving the flows or not
tcpdump -i eth0 port 3000
17:30:19.248072 IP InterfaceName.53344 > ServerName.3000: UDP, length 1464
17:30:19.248079 IP InterfaceName.53344 > ServerName.3000: UDP, length 1272
17:30:19.248853 IP InterfaceName.53344 > ServerName.3000: UDP, length 1464
17:30:19.248887 IP InterfaceName.53344 > ServerName.3000: UDP, length 1464
17:30:19.248894 IP InterfaceName.53344 > ServerName.3000: UDP, length 1272
17:30:19.249385 IP InterfaceName.60532 > ServerName.3000: UDP, length 1416
10. Now What !!!
Yes….
The Flow exporter is exporting the flows and the Flow Collector is receiving them.
So now we can start analyzingthem to understandthe traffic pattern of our network.
Can be done in many ways with many tools.
We will discuss one of the most basic tools which can be run on bash easily with little
resources and required output.
11. Flow-Tools
Flow-toolsis a library and a collection of programs used to collect, send, process, and
generate reports from NetFlow data.
Supports NetFlow version upto 8. Best output with NetFlow version 5.
Included Programs are flow-capture , flow-cat, flow-statsand many more.
http://linux.die.net/man/1/flow-tools
12. Advantages
1. Gives detailinformation on each & every particularconversationwithout sniffing.
2. No problem for encrypted data.For any incident,traffic source & Destinationsare
visible.
3. Historicaldata of Flows can help operator to improve quality.
4. Data can be fetched from anywhere in network as needed in a customized way.
5. If you are multihomed, these information are importantto make sure that your
clients are getting qualityservice.
6. NFSEN does the same work but needs bigger resources compared to Flow-Tools.
7. Ideal for startup ISPs, small enterprise, office IT network, campus network etc.
13. Flow-Tools
apt-get installflow-tools
Or get it from here and installit
https://flow-tools.googlecode.com/files/flow-tools-0.68.5.1.tar.bz2
Make a directory to store your flows
mkdir /var/flows/
14. Flow-Tools
Edit the flow-capture.conf file at /etc/flow-tools/ , comment all and use the below line.
-V 5 -E 5G -N 3 -w /var/flows 0.0.0.0/ServerIP/3000
Which means –
NetFlow Version will be 5
Expire the totalstored flow files as per the given space – here we will set 5 G
Nesting level for sorting flow files
Working directory will be /var/flows
Allow any IP as analyzer and ServerIP as exporter with port 3000
15. Flow-Tools
We can now start capturing flows with the following command.
flow-capture -w /var/flows -E5G -S3 0/0/3000
Which means –
Flow capture will start with working directory /var/flows
Totalsize of all the flow files will not exceed 5 G
Emit a stat log message every 3 minutes
Allow any IP as analyzer and exporter with port 3000
16. Flow-Tools
Now if you go to /var/flows/2015/2015-10/2015-10-27/ to see the flow files.
Filenamesbegining with tmp which are typicallyin-progress flow files from flow-capture
are not processed.
cd /var/flows/2015/2015-10/2015-10-27/ [3 level nesting for sorting files]
ls -lah
total 259M
drwxr-xr-x 2 root root 4.0K Oct 27 17:07 .
drwxr-xr-x 3 root root 4.0K Oct 27 15:24 ..
-rw-r--r-- 1 root root 36M Oct 27 16:43 ft-v05.2015-10-27.163000+0600
-rw-r--r-- 1 root root 1022K Oct 27 16:45 ft-v05.2015-10-27.164438+0600
-rw-r--r-- 1 root root 26M Oct 27 16:54 ft-v05.2015-10-27.164500+0600
-rw-r--r-- 1 root root 2.6M Oct 27 16:55 ft-v05.2015-10-27.165435+0600
-rw-r--r-- 1 root root 12M Oct 27 17:00 ft-v05.2015-10-27.165558+0600
-rw-r--r-- 1 root root 21M Oct 27 17:07 ft-v05.2015-10-27.170000+0600
-rw-r--r-- 1 root root 16M Oct 27 17:13 tmp-v05.2015-10-27.170753+0600
17. Flow-Tools
We are ready to see some outputs finally…
0 OverallSummary
1 Average packet size distribution
2 Packets per flow distribution
3 Octets per flow distribution
4 Bandwidthper flow distribution
5 UDP/TCP destinationport
6 UDP/TCP source port
7 UDP/TCP port
8 DestinationIP
9 Source IP
10 Source/DestinationIP
11 Source or DestinationIP
12 IP protocol
13 octets for flow durationplot data
14 packets for flow durationplot data
15 short summary
16 IP Next Hop
17 Input interface
18 Output interface
19 Source AS
20 DestinationAS
21 Source/DestinationAS
22 IP ToS
23 Input/OutputInterface
24 Source Prefix
25 DestinationPrefix
26 Source/DestinationPrefix
27 Exporter IP
28 Engine Id
29 Engine Type
30 Source Tag
31 DestinationTag
32 Source/DestinationTag
18. Flow-Tools
To view output in bash we need to use the below command remaining at the flow files
directory which is /var/flows/2015/2015-10/2015-10-27/
flow-cat -p ft-v05.2015-10-27.170000+0600 | flow-stat -f11 -P -p -S4 | head -30
Meaning –
Concatenateflow file named ft-v05.2015-10-27.170000+0600
The headers are preloaded for this file containingthe metadata.
Flow-stat will provide function 11 (Source or DestinationIP) with preloaded headers and
Percentage to the total amount for 4 minutes durationof flows.