Check these out next
Steven Le Roux - Kafka et Storm au service de la lutte antiDDoS à OVH - Soirée Big Data du FinistJUG
Nov. 4, 2014•0 likes•4,703 views
5 likes
Be the first to like this
Show More
views
Total views
0
On Slideshare
0
From embeds
0
Number of embeds
0
Download to read offline
Report
Technology
Kafka et Storm au service de la lutte antiDDoS à OVH Par Steven Le Roux Présentation faite à la Soirée Big Data du FinistJUG le 28/10/2014
Horacio GonzalezFollow
Spaniard lost in Brittany, unconformist coder, Java craftsman, dreamer and all-around geek at Cityzen DataRecommended
HUGFR : Une infrastructure Kafka & Storm pour lutter contre les attaques DDoS...Modern Data Stack France
zmq.rs - A brief history of concurrency in RustFantix King 王川
Kernel Recipes 2017 - EBPF and XDP - Eric LeblondAnne Nicolas
Disaggregating Ceph using NVMeoFZoltan Arnold Nagy
Data Structures for High Resolution, Real-time Telemetry at ScaleScyllaDB
4th RICC workshopのご案内Hiroki Kashiwazaki
Steven Le Roux - Kafka et Storm au service de la lutte antiDDoS à OVH - Soirée Big Data du FinistJUG
- Kafka et Storm au service de la lutte antiDDoS à OVH Steven Le Roux Infrastructure Engineer OVH.com FinistJUG 29 Oct 2014
- OVH Anti-DDoS
- VAC
- 10
- 3 Tbps 17 Datacenters 32 PoPs
- 20
- Data Pipeline
- Clients Producers Consumers Brokers Topics Partitions Replicas / kafka
- / kafka / topic
- / kafka / topic / replicas
- / kafka / topic / replicas / factor / 3
- / kafka / topics
- / kafka
- / kafka
- / kafka / producers
- Stream Processing
- Topology (DAG) Spouts Bolts Tuples Fields Cluster Nimbus Supervisors Workers / storm
- / storm / tuple field
- / storm / tuple {field1, field2,…,fieldn}
- / storm / tuple {field1, field2,…,fieldn}Bolt Bolt
- / storm / topology
- / storm / topology / antiddos
- Stream Grouping
- Shuffle Grouping Field Grouping Direct Grouping Other Grouping / storm
- Attacks Router Grouping Scans IP src Grouping / storm
- Attacks ≈ 1s Scoring Filters Burst Scans IP Proto / storm
- Indexing Prooving Producing / storm / event
- #lifecycle
- #dataviz
- Nice speech… … so what ?
- False positives Strange behaviours from customers e.g. DB sync without connection pool Application centric i.e. UDP protocols #issues
- Add other sources Application Anti-DDoS Game Half Life/Source CS:GO TeamSpeak / Mumble GTA SA:MP … More to come (any special need ?) #solutions
- #datalake
- Nodes - Hardware CPU 16c/32t RAM 256GB Disks : OS : Raid 1 Data : 10 disks per node 200 MB/s ~ 1,5-2 Gbps #hardware
- Kafka I/O bound Bench (1node) 1M+ msg/s No compression No ackers 80MB/s Tuning num.io.thread num.network.thread socket.*.buffer.* Storm CPU/RAM bound M+ tuples/s No ackers Break SRP Minimal workers Avoid transfer buffer #config
- OpenSOC
- Clément Sciascia - @csciasci Magnus Edenhill - @edenhillm https://github.com/edenhill/librdkafka LinkedIn - Apache Kafka Nathan Marz - Apache Storm #Thanks #moreStorm basic training – Mickael G. Noll http://fr.slideshare.net/miguno/apache-storm-09-basic-training-verisign Kafka documentation
- Thanks Steven LE ROUX @StevenLeRoux steven.le-roux@ovh.net