SlideShare a Scribd company logo
1 of 15
Capturing Network Traffic into
Database
Key Words: Sniffer, Network Analyzer,
Wireshark, MySQL, Database, PCAP
to MySQL
How to Store Packets into Database
(for example, MySQL)
• Having packets in database can be
very convenient:
– More performance
– Parallel writing
– Quick analysis
– Data Mining (if you want)
– Long time storage
How???

PCAP
(or real-time
capturing)

XML output

MySQL
What do we need?
• tshark (supplied with WireShark)
• PHP with XML, Xpath support
• MySQL database
My workstation
• All examples here are done in Windows
7, but if you want, you will not need a
lot of time to make them for Linux
• Our task: capture TCP packets (IP from,
IP to, port from, port to, length,
sequence) into database. Example can
be any, for instance, checking for
network scanning.
Distributed
• This can be distributed, no problem,
but you need to use extra network or
filters (otherwise, you will hang your
system: 1 sniffed packet sent make 1
more, and so on).
Distributed
Remote SQL Server

WorkStation

WorkStation
WorkStation
Getting traffic XML format
• tshark -r "D:test.pcap" -T pdml > D:test_T.xml
– Converting pcap into XML

Or
• tshark -T pdml | you_application.exe
– Real-Time
Output XML example
Warning
• Such converting to XML consume a lot of
space (50x)! PCAP file from 200 Kb grew into
10 Mb XML!!!
• In this case you might find useful to divide one
big pcap file into several of smaller size
• Also filtering is good idea, so you can throw
out fields useless for you.
XML output file structure
• It is very simple (I crossed out trivial parts, so real lines are
bigger):
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="pdml2html.xsl"?>
<pdml >
<packet>
Packet info
</packet>
<packet>
Packet info
</packet>
And so on.
XML structure – packets (Example,
DNS query)
<packet>
<proto name="geninfo" pos="0" showname="General information" size="73">
</proto>
<proto name="frame" showname="Frame 1: 73 bytes on wire (584 bits), 73 bytes
captured (584 bits)" size="73" pos="0">
</proto>
<proto name="eth">
</proto>
<proto name="ip" >
</proto>
<proto name="udp" showname="User Datagram Protocol, Src Port: 58150 (58150),
Dst Port: domain (53)" size="8" pos="34">
</proto>
<proto name="dns" showname="Domain Name System (query)" size="31" pos="42">
</proto>
</packet>
(Child elements and attributes of proto are not shown here)
XML to MySQL
• You can use LOTS of options: C++/Java,etc.
• I used SimpleXML and XPath with PHP:
$file = "test_T.xml";
$my_file = simplexml_load_file($file );
foreach ($my_file >xpath('//packet') as $packet)
{
$packet_type = $packet->proto[4];
echo $packet_type['name']; //protocol
}
And putting into database
function LoadToDataBase($con)
{
$stmt =$con->prepare("INSERT INTO tcp (capture_order, from_ip,
to_ip, from_port, to_port, tcp_length, tcp_stream, tcp_stream_text,
tcp_sequence_dec) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)");

$stmt->bind_param('sssiiiisi', $this->capture_order,$this->from_ip,
$this->to_ip,
$this->from_port,$this->to_port,
$this->tcp_length,$this->tcp_stream, $this->tcp_stream_text,
$this->tcp_sequence_dec);
$stmt->execute();
}
Here $con is open connection to mysql, and all this vars I got in cycle.
Please, refer to full code.
Thank you
• I hope you find this useful.

• Full code is available at
http://tigrantsat.me/randd/pcaptomysql/

More Related Content

What's hot

Word representation: SVD, LSA, Word2Vec
Word representation: SVD, LSA, Word2VecWord representation: SVD, LSA, Word2Vec
Word representation: SVD, LSA, Word2Vecananth
 
Intel® QuickAssist Technology Introduction, Applications, and Lab, Including ...
Intel® QuickAssist Technology Introduction, Applications, and Lab, Including ...Intel® QuickAssist Technology Introduction, Applications, and Lab, Including ...
Intel® QuickAssist Technology Introduction, Applications, and Lab, Including ...Michelle Holley
 
New Directions for Apache Arrow
New Directions for Apache ArrowNew Directions for Apache Arrow
New Directions for Apache ArrowWes McKinney
 
PostgreSQL on EXT4, XFS, BTRFS and ZFS
PostgreSQL on EXT4, XFS, BTRFS and ZFSPostgreSQL on EXT4, XFS, BTRFS and ZFS
PostgreSQL on EXT4, XFS, BTRFS and ZFSTomas Vondra
 
Maximum Overdrive: Tuning the Spark Cassandra Connector
Maximum Overdrive: Tuning the Spark Cassandra ConnectorMaximum Overdrive: Tuning the Spark Cassandra Connector
Maximum Overdrive: Tuning the Spark Cassandra ConnectorRussell Spitzer
 
Regular Expression
Regular ExpressionRegular Expression
Regular Expressionvaluebound
 
Infrastructure at Scale: Apache Kafka, Twitter Storm & Elastic Search (ARC303...
Infrastructure at Scale: Apache Kafka, Twitter Storm & Elastic Search (ARC303...Infrastructure at Scale: Apache Kafka, Twitter Storm & Elastic Search (ARC303...
Infrastructure at Scale: Apache Kafka, Twitter Storm & Elastic Search (ARC303...Amazon Web Services
 
Challenges in nlp
Challenges in nlpChallenges in nlp
Challenges in nlpZareen Syed
 
Using eBPF to Measure the k8s Cluster Health
Using eBPF to Measure the k8s Cluster HealthUsing eBPF to Measure the k8s Cluster Health
Using eBPF to Measure the k8s Cluster HealthScyllaDB
 
Hyperloglog Project
Hyperloglog ProjectHyperloglog Project
Hyperloglog ProjectKendrick Lo
 
Memory Optimization
Memory OptimizationMemory Optimization
Memory OptimizationWei Lin
 
Hortonworks Technical Workshop: Interactive Query with Apache Hive
Hortonworks Technical Workshop: Interactive Query with Apache Hive Hortonworks Technical Workshop: Interactive Query with Apache Hive
Hortonworks Technical Workshop: Interactive Query with Apache Hive Hortonworks
 
Continguous Memory Allocator in the Linux Kernel
Continguous Memory Allocator in the Linux KernelContinguous Memory Allocator in the Linux Kernel
Continguous Memory Allocator in the Linux KernelKernel TLV
 

What's hot (20)

Regular expressions
Regular expressionsRegular expressions
Regular expressions
 
Word representation: SVD, LSA, Word2Vec
Word representation: SVD, LSA, Word2VecWord representation: SVD, LSA, Word2Vec
Word representation: SVD, LSA, Word2Vec
 
04 cache memory
04 cache memory04 cache memory
04 cache memory
 
Intel® QuickAssist Technology Introduction, Applications, and Lab, Including ...
Intel® QuickAssist Technology Introduction, Applications, and Lab, Including ...Intel® QuickAssist Technology Introduction, Applications, and Lab, Including ...
Intel® QuickAssist Technology Introduction, Applications, and Lab, Including ...
 
New Directions for Apache Arrow
New Directions for Apache ArrowNew Directions for Apache Arrow
New Directions for Apache Arrow
 
PostgreSQL on EXT4, XFS, BTRFS and ZFS
PostgreSQL on EXT4, XFS, BTRFS and ZFSPostgreSQL on EXT4, XFS, BTRFS and ZFS
PostgreSQL on EXT4, XFS, BTRFS and ZFS
 
Maximum Overdrive: Tuning the Spark Cassandra Connector
Maximum Overdrive: Tuning the Spark Cassandra ConnectorMaximum Overdrive: Tuning the Spark Cassandra Connector
Maximum Overdrive: Tuning the Spark Cassandra Connector
 
Regular Expression
Regular ExpressionRegular Expression
Regular Expression
 
Hadoop Pig
Hadoop PigHadoop Pig
Hadoop Pig
 
Infrastructure at Scale: Apache Kafka, Twitter Storm & Elastic Search (ARC303...
Infrastructure at Scale: Apache Kafka, Twitter Storm & Elastic Search (ARC303...Infrastructure at Scale: Apache Kafka, Twitter Storm & Elastic Search (ARC303...
Infrastructure at Scale: Apache Kafka, Twitter Storm & Elastic Search (ARC303...
 
OVS v OVS-DPDK
OVS v OVS-DPDKOVS v OVS-DPDK
OVS v OVS-DPDK
 
Challenges in nlp
Challenges in nlpChallenges in nlp
Challenges in nlp
 
HDFS: Optimization, Stabilization and Supportability
HDFS: Optimization, Stabilization and SupportabilityHDFS: Optimization, Stabilization and Supportability
HDFS: Optimization, Stabilization and Supportability
 
Using eBPF to Measure the k8s Cluster Health
Using eBPF to Measure the k8s Cluster HealthUsing eBPF to Measure the k8s Cluster Health
Using eBPF to Measure the k8s Cluster Health
 
Hyperloglog Project
Hyperloglog ProjectHyperloglog Project
Hyperloglog Project
 
Memory Optimization
Memory OptimizationMemory Optimization
Memory Optimization
 
Hortonworks Technical Workshop: Interactive Query with Apache Hive
Hortonworks Technical Workshop: Interactive Query with Apache Hive Hortonworks Technical Workshop: Interactive Query with Apache Hive
Hortonworks Technical Workshop: Interactive Query with Apache Hive
 
Continguous Memory Allocator in the Linux Kernel
Continguous Memory Allocator in the Linux KernelContinguous Memory Allocator in the Linux Kernel
Continguous Memory Allocator in the Linux Kernel
 
Introduction to Redis
Introduction to RedisIntroduction to Redis
Introduction to Redis
 
Text categorization
Text categorizationText categorization
Text categorization
 

Viewers also liked

Ranking Merco 2014: EAE Business School, segunda escuela de negocios más repu...
Ranking Merco 2014: EAE Business School, segunda escuela de negocios más repu...Ranking Merco 2014: EAE Business School, segunda escuela de negocios más repu...
Ranking Merco 2014: EAE Business School, segunda escuela de negocios más repu...EAE Business School
 
Open source network forensics and advanced pcap analysis
Open source network forensics and advanced pcap analysisOpen source network forensics and advanced pcap analysis
Open source network forensics and advanced pcap analysisGTKlondike
 
MySQL/MariaDB query optimizer tuning tutorial from Percona Live 2013
MySQL/MariaDB query optimizer tuning tutorial from Percona Live 2013MySQL/MariaDB query optimizer tuning tutorial from Percona Live 2013
MySQL/MariaDB query optimizer tuning tutorial from Percona Live 2013Sergey Petrunya
 
CapAnalysis - Deep Packet Inspection
CapAnalysis - Deep Packet InspectionCapAnalysis - Deep Packet Inspection
CapAnalysis - Deep Packet InspectionChris Harrington
 
Wireshar training
Wireshar trainingWireshar training
Wireshar trainingLuke Luo
 
Authentication, authorization, accounting(aaa) slides
Authentication, authorization, accounting(aaa) slidesAuthentication, authorization, accounting(aaa) slides
Authentication, authorization, accounting(aaa) slidesrahul kundu
 
Radius server,PAP and CHAP Protocols
Radius server,PAP and CHAP ProtocolsRadius server,PAP and CHAP Protocols
Radius server,PAP and CHAP ProtocolsDhananjay Aloorkar
 
Implementing Cisco AAA
Implementing Cisco AAAImplementing Cisco AAA
Implementing Cisco AAAdkaya
 
What is PCRF? – Detailed PCRF architecture and functioning
What is PCRF? – Detailed PCRF architecture and functioningWhat is PCRF? – Detailed PCRF architecture and functioning
What is PCRF? – Detailed PCRF architecture and functioningMahindra Comviva
 
Wireshark
WiresharkWireshark
Wiresharkbtohara
 
Radius Protocol
Radius ProtocolRadius Protocol
Radius ProtocolNetwax Lab
 
AAA & RADIUS Protocols
AAA & RADIUS ProtocolsAAA & RADIUS Protocols
AAA & RADIUS ProtocolsPeter R. Egli
 

Viewers also liked (20)

Traffic types in internet
Traffic types in internetTraffic types in internet
Traffic types in internet
 
Ranking Merco 2014: EAE Business School, segunda escuela de negocios más repu...
Ranking Merco 2014: EAE Business School, segunda escuela de negocios más repu...Ranking Merco 2014: EAE Business School, segunda escuela de negocios más repu...
Ranking Merco 2014: EAE Business School, segunda escuela de negocios más repu...
 
Open source network forensics and advanced pcap analysis
Open source network forensics and advanced pcap analysisOpen source network forensics and advanced pcap analysis
Open source network forensics and advanced pcap analysis
 
MySQL/MariaDB query optimizer tuning tutorial from Percona Live 2013
MySQL/MariaDB query optimizer tuning tutorial from Percona Live 2013MySQL/MariaDB query optimizer tuning tutorial from Percona Live 2013
MySQL/MariaDB query optimizer tuning tutorial from Percona Live 2013
 
CapAnalysis - Deep Packet Inspection
CapAnalysis - Deep Packet InspectionCapAnalysis - Deep Packet Inspection
CapAnalysis - Deep Packet Inspection
 
AfterGlow
AfterGlowAfterGlow
AfterGlow
 
Wireshar training
Wireshar trainingWireshar training
Wireshar training
 
AAA in a nutshell
AAA in a nutshellAAA in a nutshell
AAA in a nutshell
 
Authentication, authorization, accounting(aaa) slides
Authentication, authorization, accounting(aaa) slidesAuthentication, authorization, accounting(aaa) slides
Authentication, authorization, accounting(aaa) slides
 
Radius server,PAP and CHAP Protocols
Radius server,PAP and CHAP ProtocolsRadius server,PAP and CHAP Protocols
Radius server,PAP and CHAP Protocols
 
RADIUS
RADIUSRADIUS
RADIUS
 
Quality of Service
Quality of ServiceQuality of Service
Quality of Service
 
Implementing Cisco AAA
Implementing Cisco AAAImplementing Cisco AAA
Implementing Cisco AAA
 
What is PCRF? – Detailed PCRF architecture and functioning
What is PCRF? – Detailed PCRF architecture and functioningWhat is PCRF? – Detailed PCRF architecture and functioning
What is PCRF? – Detailed PCRF architecture and functioning
 
AAA Protocol
AAA ProtocolAAA Protocol
AAA Protocol
 
Wireshark
WiresharkWireshark
Wireshark
 
Introduction to Diameter Protocol - Part1
Introduction to Diameter Protocol - Part1Introduction to Diameter Protocol - Part1
Introduction to Diameter Protocol - Part1
 
Radius Protocol
Radius ProtocolRadius Protocol
Radius Protocol
 
Wireshark
WiresharkWireshark
Wireshark
 
AAA & RADIUS Protocols
AAA & RADIUS ProtocolsAAA & RADIUS Protocols
AAA & RADIUS Protocols
 

Similar to Capturing Network Traffic into Database

Project Tungsten Phase II: Joining a Billion Rows per Second on a Laptop
Project Tungsten Phase II: Joining a Billion Rows per Second on a LaptopProject Tungsten Phase II: Joining a Billion Rows per Second on a Laptop
Project Tungsten Phase II: Joining a Billion Rows per Second on a LaptopDatabricks
 
High Performance, Scalable MongoDB in a Bare Metal Cloud
High Performance, Scalable MongoDB in a Bare Metal CloudHigh Performance, Scalable MongoDB in a Bare Metal Cloud
High Performance, Scalable MongoDB in a Bare Metal CloudMongoDB
 
User-space Network Processing
User-space Network ProcessingUser-space Network Processing
User-space Network ProcessingRyousei Takano
 
High performace network of Cloud Native Taiwan User Group
High performace network of Cloud Native Taiwan User GroupHigh performace network of Cloud Native Taiwan User Group
High performace network of Cloud Native Taiwan User GroupHungWei Chiu
 
Attack monitoring using ElasticSearch Logstash and Kibana
Attack monitoring using ElasticSearch Logstash and KibanaAttack monitoring using ElasticSearch Logstash and Kibana
Attack monitoring using ElasticSearch Logstash and KibanaPrajal Kulkarni
 
Ga4 gh meeting at the the sanger institute
Ga4 gh meeting at the the sanger instituteGa4 gh meeting at the the sanger institute
Ga4 gh meeting at the the sanger instituteMatt Massie
 
Explore big data at speed of thought with Spark 2.0 and Snappydata
Explore big data at speed of thought with Spark 2.0 and SnappydataExplore big data at speed of thought with Spark 2.0 and Snappydata
Explore big data at speed of thought with Spark 2.0 and SnappydataData Con LA
 
Project Tungsten: Bringing Spark Closer to Bare Metal
Project Tungsten: Bringing Spark Closer to Bare MetalProject Tungsten: Bringing Spark Closer to Bare Metal
Project Tungsten: Bringing Spark Closer to Bare MetalDatabricks
 
Event Processing and Integration with IAS Data Processors
Event Processing and Integration with IAS Data ProcessorsEvent Processing and Integration with IAS Data Processors
Event Processing and Integration with IAS Data ProcessorsInvenire Aude
 
End to End Processing of 3.7 Million Telemetry Events per Second using Lambda...
End to End Processing of 3.7 Million Telemetry Events per Second using Lambda...End to End Processing of 3.7 Million Telemetry Events per Second using Lambda...
End to End Processing of 3.7 Million Telemetry Events per Second using Lambda...DataWorks Summit/Hadoop Summit
 
osdi20-slides_zhao.pptx
osdi20-slides_zhao.pptxosdi20-slides_zhao.pptx
osdi20-slides_zhao.pptxCive1971
 
Hardware Provisioning
Hardware ProvisioningHardware Provisioning
Hardware ProvisioningMongoDB
 
KSQL Performance Tuning for Fun and Profit ( Nick Dearden, Confluent) Kafka S...
KSQL Performance Tuning for Fun and Profit ( Nick Dearden, Confluent) Kafka S...KSQL Performance Tuning for Fun and Profit ( Nick Dearden, Confluent) Kafka S...
KSQL Performance Tuning for Fun and Profit ( Nick Dearden, Confluent) Kafka S...confluent
 
DPDK Summit 2015 - Aspera - Charles Shiflett
DPDK Summit 2015 - Aspera - Charles ShiflettDPDK Summit 2015 - Aspera - Charles Shiflett
DPDK Summit 2015 - Aspera - Charles ShiflettJim St. Leger
 
Practical 7 - Using Wireshark Tutorial and Hands-on
Practical 7 - Using Wireshark Tutorial and Hands-onPractical 7 - Using Wireshark Tutorial and Hands-on
Practical 7 - Using Wireshark Tutorial and Hands-onQaisSaifQassim
 
seed block algorithm
seed block algorithmseed block algorithm
seed block algorithmDipak Badhe
 
Multitenancy: Kafka clusters for everyone at LINE
Multitenancy: Kafka clusters for everyone at LINEMultitenancy: Kafka clusters for everyone at LINE
Multitenancy: Kafka clusters for everyone at LINEkawamuray
 
Python (Jinja2) Templates for Network Automation
Python (Jinja2) Templates for Network AutomationPython (Jinja2) Templates for Network Automation
Python (Jinja2) Templates for Network AutomationRick Sherman
 
Big data analytics with Spark & Cassandra
Big data analytics with Spark & Cassandra Big data analytics with Spark & Cassandra
Big data analytics with Spark & Cassandra Matthias Niehoff
 

Similar to Capturing Network Traffic into Database (20)

Project Tungsten Phase II: Joining a Billion Rows per Second on a Laptop
Project Tungsten Phase II: Joining a Billion Rows per Second on a LaptopProject Tungsten Phase II: Joining a Billion Rows per Second on a Laptop
Project Tungsten Phase II: Joining a Billion Rows per Second on a Laptop
 
High Performance, Scalable MongoDB in a Bare Metal Cloud
High Performance, Scalable MongoDB in a Bare Metal CloudHigh Performance, Scalable MongoDB in a Bare Metal Cloud
High Performance, Scalable MongoDB in a Bare Metal Cloud
 
User-space Network Processing
User-space Network ProcessingUser-space Network Processing
User-space Network Processing
 
High performace network of Cloud Native Taiwan User Group
High performace network of Cloud Native Taiwan User GroupHigh performace network of Cloud Native Taiwan User Group
High performace network of Cloud Native Taiwan User Group
 
Jumbo Mumbo in OpenStack
Jumbo Mumbo in OpenStackJumbo Mumbo in OpenStack
Jumbo Mumbo in OpenStack
 
Attack monitoring using ElasticSearch Logstash and Kibana
Attack monitoring using ElasticSearch Logstash and KibanaAttack monitoring using ElasticSearch Logstash and Kibana
Attack monitoring using ElasticSearch Logstash and Kibana
 
Ga4 gh meeting at the the sanger institute
Ga4 gh meeting at the the sanger instituteGa4 gh meeting at the the sanger institute
Ga4 gh meeting at the the sanger institute
 
Explore big data at speed of thought with Spark 2.0 and Snappydata
Explore big data at speed of thought with Spark 2.0 and SnappydataExplore big data at speed of thought with Spark 2.0 and Snappydata
Explore big data at speed of thought with Spark 2.0 and Snappydata
 
Project Tungsten: Bringing Spark Closer to Bare Metal
Project Tungsten: Bringing Spark Closer to Bare MetalProject Tungsten: Bringing Spark Closer to Bare Metal
Project Tungsten: Bringing Spark Closer to Bare Metal
 
Event Processing and Integration with IAS Data Processors
Event Processing and Integration with IAS Data ProcessorsEvent Processing and Integration with IAS Data Processors
Event Processing and Integration with IAS Data Processors
 
End to End Processing of 3.7 Million Telemetry Events per Second using Lambda...
End to End Processing of 3.7 Million Telemetry Events per Second using Lambda...End to End Processing of 3.7 Million Telemetry Events per Second using Lambda...
End to End Processing of 3.7 Million Telemetry Events per Second using Lambda...
 
osdi20-slides_zhao.pptx
osdi20-slides_zhao.pptxosdi20-slides_zhao.pptx
osdi20-slides_zhao.pptx
 
Hardware Provisioning
Hardware ProvisioningHardware Provisioning
Hardware Provisioning
 
KSQL Performance Tuning for Fun and Profit ( Nick Dearden, Confluent) Kafka S...
KSQL Performance Tuning for Fun and Profit ( Nick Dearden, Confluent) Kafka S...KSQL Performance Tuning for Fun and Profit ( Nick Dearden, Confluent) Kafka S...
KSQL Performance Tuning for Fun and Profit ( Nick Dearden, Confluent) Kafka S...
 
DPDK Summit 2015 - Aspera - Charles Shiflett
DPDK Summit 2015 - Aspera - Charles ShiflettDPDK Summit 2015 - Aspera - Charles Shiflett
DPDK Summit 2015 - Aspera - Charles Shiflett
 
Practical 7 - Using Wireshark Tutorial and Hands-on
Practical 7 - Using Wireshark Tutorial and Hands-onPractical 7 - Using Wireshark Tutorial and Hands-on
Practical 7 - Using Wireshark Tutorial and Hands-on
 
seed block algorithm
seed block algorithmseed block algorithm
seed block algorithm
 
Multitenancy: Kafka clusters for everyone at LINE
Multitenancy: Kafka clusters for everyone at LINEMultitenancy: Kafka clusters for everyone at LINE
Multitenancy: Kafka clusters for everyone at LINE
 
Python (Jinja2) Templates for Network Automation
Python (Jinja2) Templates for Network AutomationPython (Jinja2) Templates for Network Automation
Python (Jinja2) Templates for Network Automation
 
Big data analytics with Spark & Cassandra
Big data analytics with Spark & Cassandra Big data analytics with Spark & Cassandra
Big data analytics with Spark & Cassandra
 

Recently uploaded

Introduction to use of FHIR Documents in ABDM
Introduction to use of FHIR Documents in ABDMIntroduction to use of FHIR Documents in ABDM
Introduction to use of FHIR Documents in ABDMKumar Satyam
 
WSO2 Micro Integrator for Enterprise Integration in a Decentralized, Microser...
WSO2 Micro Integrator for Enterprise Integration in a Decentralized, Microser...WSO2 Micro Integrator for Enterprise Integration in a Decentralized, Microser...
WSO2 Micro Integrator for Enterprise Integration in a Decentralized, Microser...WSO2
 
Simplifying Mobile A11y Presentation.pptx
Simplifying Mobile A11y Presentation.pptxSimplifying Mobile A11y Presentation.pptx
Simplifying Mobile A11y Presentation.pptxMarkSteadman7
 
Quantum Leap in Next-Generation Computing
Quantum Leap in Next-Generation ComputingQuantum Leap in Next-Generation Computing
Quantum Leap in Next-Generation ComputingWSO2
 
Vector Search -An Introduction in Oracle Database 23ai.pptx
Vector Search -An Introduction in Oracle Database 23ai.pptxVector Search -An Introduction in Oracle Database 23ai.pptx
Vector Search -An Introduction in Oracle Database 23ai.pptxRemote DBA Services
 
Stronger Together: Developing an Organizational Strategy for Accessible Desig...
Stronger Together: Developing an Organizational Strategy for Accessible Desig...Stronger Together: Developing an Organizational Strategy for Accessible Desig...
Stronger Together: Developing an Organizational Strategy for Accessible Desig...caitlingebhard1
 
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...DianaGray10
 
Architecting Cloud Native Applications
Architecting Cloud Native ApplicationsArchitecting Cloud Native Applications
Architecting Cloud Native ApplicationsWSO2
 
Decarbonising Commercial Real Estate: The Role of Operational Performance
Decarbonising Commercial Real Estate: The Role of Operational PerformanceDecarbonising Commercial Real Estate: The Role of Operational Performance
Decarbonising Commercial Real Estate: The Role of Operational PerformanceIES VE
 
Navigating Identity and Access Management in the Modern Enterprise
Navigating Identity and Access Management in the Modern EnterpriseNavigating Identity and Access Management in the Modern Enterprise
Navigating Identity and Access Management in the Modern EnterpriseWSO2
 
AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)
AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)
AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)Samir Dash
 
JohnPollard-hybrid-app-RailsConf2024.pptx
JohnPollard-hybrid-app-RailsConf2024.pptxJohnPollard-hybrid-app-RailsConf2024.pptx
JohnPollard-hybrid-app-RailsConf2024.pptxJohnPollard37
 
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdfRising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdfOrbitshub
 
Platformless Horizons for Digital Adaptability
Platformless Horizons for Digital AdaptabilityPlatformless Horizons for Digital Adaptability
Platformless Horizons for Digital AdaptabilityWSO2
 
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FMESafe Software
 
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost SavingRepurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost SavingEdi Saputra
 
TEST BANK For Principles of Anatomy and Physiology, 16th Edition by Gerard J....
TEST BANK For Principles of Anatomy and Physiology, 16th Edition by Gerard J....TEST BANK For Principles of Anatomy and Physiology, 16th Edition by Gerard J....
TEST BANK For Principles of Anatomy and Physiology, 16th Edition by Gerard J....rightmanforbloodline
 
Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024Victor Rentea
 
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...Jeffrey Haguewood
 

Recently uploaded (20)

Introduction to use of FHIR Documents in ABDM
Introduction to use of FHIR Documents in ABDMIntroduction to use of FHIR Documents in ABDM
Introduction to use of FHIR Documents in ABDM
 
WSO2 Micro Integrator for Enterprise Integration in a Decentralized, Microser...
WSO2 Micro Integrator for Enterprise Integration in a Decentralized, Microser...WSO2 Micro Integrator for Enterprise Integration in a Decentralized, Microser...
WSO2 Micro Integrator for Enterprise Integration in a Decentralized, Microser...
 
Simplifying Mobile A11y Presentation.pptx
Simplifying Mobile A11y Presentation.pptxSimplifying Mobile A11y Presentation.pptx
Simplifying Mobile A11y Presentation.pptx
 
Quantum Leap in Next-Generation Computing
Quantum Leap in Next-Generation ComputingQuantum Leap in Next-Generation Computing
Quantum Leap in Next-Generation Computing
 
Vector Search -An Introduction in Oracle Database 23ai.pptx
Vector Search -An Introduction in Oracle Database 23ai.pptxVector Search -An Introduction in Oracle Database 23ai.pptx
Vector Search -An Introduction in Oracle Database 23ai.pptx
 
Stronger Together: Developing an Organizational Strategy for Accessible Desig...
Stronger Together: Developing an Organizational Strategy for Accessible Desig...Stronger Together: Developing an Organizational Strategy for Accessible Desig...
Stronger Together: Developing an Organizational Strategy for Accessible Desig...
 
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
 
Architecting Cloud Native Applications
Architecting Cloud Native ApplicationsArchitecting Cloud Native Applications
Architecting Cloud Native Applications
 
Decarbonising Commercial Real Estate: The Role of Operational Performance
Decarbonising Commercial Real Estate: The Role of Operational PerformanceDecarbonising Commercial Real Estate: The Role of Operational Performance
Decarbonising Commercial Real Estate: The Role of Operational Performance
 
Navigating Identity and Access Management in the Modern Enterprise
Navigating Identity and Access Management in the Modern EnterpriseNavigating Identity and Access Management in the Modern Enterprise
Navigating Identity and Access Management in the Modern Enterprise
 
AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)
AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)
AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)
 
JohnPollard-hybrid-app-RailsConf2024.pptx
JohnPollard-hybrid-app-RailsConf2024.pptxJohnPollard-hybrid-app-RailsConf2024.pptx
JohnPollard-hybrid-app-RailsConf2024.pptx
 
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdfRising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf
Rising Above_ Dubai Floods and the Fortitude of Dubai International Airport.pdf
 
Platformless Horizons for Digital Adaptability
Platformless Horizons for Digital AdaptabilityPlatformless Horizons for Digital Adaptability
Platformless Horizons for Digital Adaptability
 
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
 
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost SavingRepurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
 
TEST BANK For Principles of Anatomy and Physiology, 16th Edition by Gerard J....
TEST BANK For Principles of Anatomy and Physiology, 16th Edition by Gerard J....TEST BANK For Principles of Anatomy and Physiology, 16th Edition by Gerard J....
TEST BANK For Principles of Anatomy and Physiology, 16th Edition by Gerard J....
 
Understanding the FAA Part 107 License ..
Understanding the FAA Part 107 License ..Understanding the FAA Part 107 License ..
Understanding the FAA Part 107 License ..
 
Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024
 
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
Web Form Automation for Bonterra Impact Management (fka Social Solutions Apri...
 

Capturing Network Traffic into Database

  • 1. Capturing Network Traffic into Database Key Words: Sniffer, Network Analyzer, Wireshark, MySQL, Database, PCAP to MySQL
  • 2. How to Store Packets into Database (for example, MySQL) • Having packets in database can be very convenient: – More performance – Parallel writing – Quick analysis – Data Mining (if you want) – Long time storage
  • 4. What do we need? • tshark (supplied with WireShark) • PHP with XML, Xpath support • MySQL database
  • 5. My workstation • All examples here are done in Windows 7, but if you want, you will not need a lot of time to make them for Linux • Our task: capture TCP packets (IP from, IP to, port from, port to, length, sequence) into database. Example can be any, for instance, checking for network scanning.
  • 6. Distributed • This can be distributed, no problem, but you need to use extra network or filters (otherwise, you will hang your system: 1 sniffed packet sent make 1 more, and so on).
  • 8. Getting traffic XML format • tshark -r "D:test.pcap" -T pdml > D:test_T.xml – Converting pcap into XML Or • tshark -T pdml | you_application.exe – Real-Time
  • 10. Warning • Such converting to XML consume a lot of space (50x)! PCAP file from 200 Kb grew into 10 Mb XML!!! • In this case you might find useful to divide one big pcap file into several of smaller size • Also filtering is good idea, so you can throw out fields useless for you.
  • 11. XML output file structure • It is very simple (I crossed out trivial parts, so real lines are bigger): <?xml version="1.0"?> <?xml-stylesheet type="text/xsl" href="pdml2html.xsl"?> <pdml > <packet> Packet info </packet> <packet> Packet info </packet> And so on.
  • 12. XML structure – packets (Example, DNS query) <packet> <proto name="geninfo" pos="0" showname="General information" size="73"> </proto> <proto name="frame" showname="Frame 1: 73 bytes on wire (584 bits), 73 bytes captured (584 bits)" size="73" pos="0"> </proto> <proto name="eth"> </proto> <proto name="ip" > </proto> <proto name="udp" showname="User Datagram Protocol, Src Port: 58150 (58150), Dst Port: domain (53)" size="8" pos="34"> </proto> <proto name="dns" showname="Domain Name System (query)" size="31" pos="42"> </proto> </packet> (Child elements and attributes of proto are not shown here)
  • 13. XML to MySQL • You can use LOTS of options: C++/Java,etc. • I used SimpleXML and XPath with PHP: $file = "test_T.xml"; $my_file = simplexml_load_file($file ); foreach ($my_file >xpath('//packet') as $packet) { $packet_type = $packet->proto[4]; echo $packet_type['name']; //protocol }
  • 14. And putting into database function LoadToDataBase($con) { $stmt =$con->prepare("INSERT INTO tcp (capture_order, from_ip, to_ip, from_port, to_port, tcp_length, tcp_stream, tcp_stream_text, tcp_sequence_dec) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)"); $stmt->bind_param('sssiiiisi', $this->capture_order,$this->from_ip, $this->to_ip, $this->from_port,$this->to_port, $this->tcp_length,$this->tcp_stream, $this->tcp_stream_text, $this->tcp_sequence_dec); $stmt->execute(); } Here $con is open connection to mysql, and all this vars I got in cycle. Please, refer to full code.
  • 15. Thank you • I hope you find this useful. • Full code is available at http://tigrantsat.me/randd/pcaptomysql/