Quantum - Virtual networks for Openstacksalv_orlando
An overview of Quantum, the soon-to-be default Openstack network service.
These slides introduce Quantum, its design goals, and discusses the API. It also tries to address how quantum relates to Software Defined Networking (SDN)
A tour of scalability improvements between Havana and Juno.
The presentation discusses results from an experimental campaign and the various features that enable the scalability improvements
Presentation from Aaron Rose and Salvatore Orlando.
Quantum - Virtual networks for Openstacksalv_orlando
An overview of Quantum, the soon-to-be default Openstack network service.
These slides introduce Quantum, its design goals, and discusses the API. It also tries to address how quantum relates to Software Defined Networking (SDN)
A tour of scalability improvements between Havana and Juno.
The presentation discusses results from an experimental campaign and the various features that enable the scalability improvements
Presentation from Aaron Rose and Salvatore Orlando.
" Breaking Extreme Networks WingOS: How to own millions of devices running on...PROIDEA
Extreme network's embedded WingOS (Originally created by Motorola) is an operating system used in several wireless devices such as access points and controllers. This OS is being used in Motorola devices, Zebra devices and Extreme network's devices. This research started focusing in an access point widely used in many Aircrafts by several worldwide airlines but ended up in something bigger in terms of devices affected as this embedded operating system is not only used in AP's for Aircrafts but also in Healthcare, Government, Transportation, Smart cities, small to big enterprises... and more. Based on public information, we will see how vulnerable devices are actively used (outdoors) in big cities around the world. But also in Universities, Hotels,Casinos, Big companies, Mines, Hospitals and provides the Wi-Fi access for places such as the New york City Subway. In this presentation we will show with technical details how several critical vulnerabilities were found in this embedded OS. First we will introduce some internals and details about the OS and then we will show the techniques used to reverse engineering the mipsN32 ABI code for the Cavium Octeon processor. It will be discussed how some code was emulated to detect how a dynamic password is generated with a cryptographic algorithm for a root shell backdoor. Besides, it will be shown how some protocols used by some services were reverse engineered to find unauthenticated heap and stack overflow vulnerabilities that could be exploitable trough Wireless or Ethernet connection. This OS also uses a proprietary layer 2/3 protocol called MiNT. This protocol is used for communication between WingOS devices through VLAN or IP. This protocol was also reverse engineered and remote heap/stack overflow vulnerabilities were found on services using this protocol and will be shown. As a demonstration, 2 devices will be used to exploit a remote stack overflow chaining several vulnerabilities as the attacker could do inside an aircraft (or other scenarios) through the Wi-Fi. As there are not public shellcodes for mipsN32 ABI, the particularities of creating a Shellcode for mipsN32 ABI will be also discussed.
Docker is the new kool kid in town. This presentation covers some of the common goof-ups and what should be kept in mind when dealing with docker configurations.
Download the Vulnerable Docker VM : https://www.notsosecure.com/vulnerable-docker-vm/
IPv6 is slowly making its way into our environments and we need to be aware of how it impacts the systems we manage. This presentation takes us through a basic review of the protocol from a pentesters perspective
Buffer overflow exploitation without operating system protections is a well understood subject. But how does one achieve the same results with all protections enabled (N/X, ASLR, …). Hint: re-use what the vulnerable binary offers you.
This presentation was shown at the OpenStack Online Meetup session on August 28, 2014. It is an update to the 2013 sessions, and adds content on Services Plugin, Modular plugins, as well as an Outlook to some Juno features like DVR, HA and IPv6 Support
Is OpenStack Neutron production ready for large scale deployments?Елена Ежова
OpenStack Neutron with ML2 OVS has always been a challenging component in terms of performance and scalability. However, in recent releases, several enhancements and bug-fixes have resulted in significant improvements in overall reliability, performance and scalability of Neutron. In this presentation, we will share the results of our testing (both control-plane and data-plane) at large scale and provide a detailed data-driven analysis that explores the true scale limits and bottlenecks of Neutron.
Red Hat demo of OpenStack and ODL at ODL summit 2016 RedHatTelco
Red Hat demonstrated OpenDaylight (ODL) as an SDN Controller for OpenStack. We showed the integration of the Boron release of OpenDaylight with the Mitaka release of OpenStack. The primary objective of the demo was to show how NetVirt can easily create and manage virtual networks that are flexible, secure and scalable.
OpenStack Neutron Advanced Services by AkandaSean Roberts
Sean Roberts, VP Development Akanda, gave this talk on 03 September 2015 at the HP Sunnyvale offices. This talk goes into detail of how Akanda delivers OpenStack Neutron Advanced Services. Event details can be found here http://www.meetup.com/openstack/events/215648162/
IKE is the protocol used to set up a security association (SA) in the IPsec protocol suite. IKEv2 is the
second and latest version of the IKE protocol. Adoption for this protocol started as early as 2006.
IKE builds upon the Oakley protocol and ISAKMP. IKE uses X.509 certificates for authentication - either
pre-shared or distributed using DNS (preferably with DNSSEC) and a Diffie–Hellman key exchange - to
set up a shared session secret from which cryptographic keys are derived.
Future Internet Week - IPv6 the way forward: IPv6 and security from a user’s ...ir. Carmelo Zaccone
This workshop will start with a presentation of results of a study that was conducted for the European Commission on IPv6 and security. This will be followed by presentations from a technology provider who will focus on the security issues related to IPv6. The last presentation will be done by an organisation that has implemented IPv6 and it will share its experiences with the focus on security. At the end of the session, there is a Q&A.
http://ipv6-ghent.fi-week.eu/ipv6-security/
" Breaking Extreme Networks WingOS: How to own millions of devices running on...PROIDEA
Extreme network's embedded WingOS (Originally created by Motorola) is an operating system used in several wireless devices such as access points and controllers. This OS is being used in Motorola devices, Zebra devices and Extreme network's devices. This research started focusing in an access point widely used in many Aircrafts by several worldwide airlines but ended up in something bigger in terms of devices affected as this embedded operating system is not only used in AP's for Aircrafts but also in Healthcare, Government, Transportation, Smart cities, small to big enterprises... and more. Based on public information, we will see how vulnerable devices are actively used (outdoors) in big cities around the world. But also in Universities, Hotels,Casinos, Big companies, Mines, Hospitals and provides the Wi-Fi access for places such as the New york City Subway. In this presentation we will show with technical details how several critical vulnerabilities were found in this embedded OS. First we will introduce some internals and details about the OS and then we will show the techniques used to reverse engineering the mipsN32 ABI code for the Cavium Octeon processor. It will be discussed how some code was emulated to detect how a dynamic password is generated with a cryptographic algorithm for a root shell backdoor. Besides, it will be shown how some protocols used by some services were reverse engineered to find unauthenticated heap and stack overflow vulnerabilities that could be exploitable trough Wireless or Ethernet connection. This OS also uses a proprietary layer 2/3 protocol called MiNT. This protocol is used for communication between WingOS devices through VLAN or IP. This protocol was also reverse engineered and remote heap/stack overflow vulnerabilities were found on services using this protocol and will be shown. As a demonstration, 2 devices will be used to exploit a remote stack overflow chaining several vulnerabilities as the attacker could do inside an aircraft (or other scenarios) through the Wi-Fi. As there are not public shellcodes for mipsN32 ABI, the particularities of creating a Shellcode for mipsN32 ABI will be also discussed.
Docker is the new kool kid in town. This presentation covers some of the common goof-ups and what should be kept in mind when dealing with docker configurations.
Download the Vulnerable Docker VM : https://www.notsosecure.com/vulnerable-docker-vm/
IPv6 is slowly making its way into our environments and we need to be aware of how it impacts the systems we manage. This presentation takes us through a basic review of the protocol from a pentesters perspective
Buffer overflow exploitation without operating system protections is a well understood subject. But how does one achieve the same results with all protections enabled (N/X, ASLR, …). Hint: re-use what the vulnerable binary offers you.
This presentation was shown at the OpenStack Online Meetup session on August 28, 2014. It is an update to the 2013 sessions, and adds content on Services Plugin, Modular plugins, as well as an Outlook to some Juno features like DVR, HA and IPv6 Support
Is OpenStack Neutron production ready for large scale deployments?Елена Ежова
OpenStack Neutron with ML2 OVS has always been a challenging component in terms of performance and scalability. However, in recent releases, several enhancements and bug-fixes have resulted in significant improvements in overall reliability, performance and scalability of Neutron. In this presentation, we will share the results of our testing (both control-plane and data-plane) at large scale and provide a detailed data-driven analysis that explores the true scale limits and bottlenecks of Neutron.
Red Hat demo of OpenStack and ODL at ODL summit 2016 RedHatTelco
Red Hat demonstrated OpenDaylight (ODL) as an SDN Controller for OpenStack. We showed the integration of the Boron release of OpenDaylight with the Mitaka release of OpenStack. The primary objective of the demo was to show how NetVirt can easily create and manage virtual networks that are flexible, secure and scalable.
OpenStack Neutron Advanced Services by AkandaSean Roberts
Sean Roberts, VP Development Akanda, gave this talk on 03 September 2015 at the HP Sunnyvale offices. This talk goes into detail of how Akanda delivers OpenStack Neutron Advanced Services. Event details can be found here http://www.meetup.com/openstack/events/215648162/
IKE is the protocol used to set up a security association (SA) in the IPsec protocol suite. IKEv2 is the
second and latest version of the IKE protocol. Adoption for this protocol started as early as 2006.
IKE builds upon the Oakley protocol and ISAKMP. IKE uses X.509 certificates for authentication - either
pre-shared or distributed using DNS (preferably with DNSSEC) and a Diffie–Hellman key exchange - to
set up a shared session secret from which cryptographic keys are derived.
Future Internet Week - IPv6 the way forward: IPv6 and security from a user’s ...ir. Carmelo Zaccone
This workshop will start with a presentation of results of a study that was conducted for the European Commission on IPv6 and security. This will be followed by presentations from a technology provider who will focus on the security issues related to IPv6. The last presentation will be done by an organisation that has implemented IPv6 and it will share its experiences with the focus on security. At the end of the session, there is a Q&A.
http://ipv6-ghent.fi-week.eu/ipv6-security/
DevSecCon London 2018: Get rid of these TLS certificatesDevSecCon
Paweł Krawczyk
Most network services and daemons now offer TLS transport protection and their managing certificates and TLS configuration for server farms may use more resources than actual configuration of these services. What if you could get rid of all this complexity and replace it by single transport protection protocol, securing all of the traffic between your servers trasparently and with single centralized key and configuration management? This will be a story of a successful implementation of IPSec protocols, largely and undeservedly forgotten in that purpose, for securing a farm of production cloud servers, with configuration centrally managed with Ansible.
Having developed a test set, we started to research how safe it is for clients to use 4G networks of the telecommunication companies. During the research we have tested SIM-cards, 4G USB modems, radio components, IP access network. First of all we looked for the vulnerabilities that could be exploited remotely, via IP or radio network.
And the result was not late in arriving. In some cases we managed to attack SIM-cards and install a malicious Java applet there, we were able to update remotely USB modem firmware, to change password on a selfcare portal via SMS and even to get access to the internal technological network of a carrier.
Further attack evolution helped to understand how it is possible to use a simple SMS as an exploit that is able not only to compromise a USB modem and all the communications that go through it, but also to install bootkit on a box, that this modem is connected to.
Как мы взломали распределенные системы конфигурационного управленияPositive Hack Days
В лекции речь пойдет о том, как команда исследователей обнаружила и эксплуатировала уязвимости различных систем конфигурационного управления в ходе пентестов. Авторы представят различные инструменты распределенного управления конфигурациями, например Apache ZooKeeper, HashiCorp Consul и Serf, CoreOS Etcd; расскажут о способах создания отпечатков этих систем, а также о том, как использовать в своих целях типичные ошибки в конфигурации для увеличения площади атак.
Aspekte von IPv6-Security
• Hackertools & ein paar Angriffsszenarien
• 3 Empfehlungen
q a) Ist IPv6 sicherer als IPv4?
q b) Ist IPv6 unsicherer als IPv4?
q c) Wer ist an allem Schuld?
q d) Wie wirkt sich die Integration von IPv6 in
meine Organisation auf deren IT-Sicherheit aus?
Pluggable Infrastructure with CI/CD and DockerBob Killen
The docker cluster ecosystem is still young, and highly modular. This presentation covers some of the challenges we faced deciding on what infrastructure to deploy, and a few tips and tricks in making both applications and infrastructure easily adaptable.
If you going to build services in China's AWS, learn from our experience.
Slides from meetup:
https://www.meetup.com/SF-DevOps-for-Startups/events/238642366/
I was asked to talk in front of Computer science students at the Bar-Ilan university about "what happens" when you don't care about writing "secured" or "safe" code. A perfect example for that, in my opinion, was the world of embedded computing AKA the IoT. I talked about the history of consumer embedded devices and showed a live demo of an 0day I found in one of the most popular routers in the country.
Similar to SREcon Europe 2016 - Full-mesh IPsec network at Hosted Graphite (20)
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf91mobiles
91mobiles recently conducted a Smart TV Buyer Insights Survey in which we asked over 3,000 respondents about the TV they own, aspects they look at on a new TV, and their TV buying preferences.
Securing your Kubernetes cluster_ a step-by-step guide to success !KatiaHIMEUR1
Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster.
However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks.
In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...UiPathCommunity
💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™:
See how to accelerate model training and optimize model performance with active learning
Learn about the latest enhancements to out-of-the-box document processing – with little to no training required
Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages
This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath.
Speakers:
👨🏫 Andras Palfi, Senior Product Manager, UiPath
👩🏫 Lenka Dulovicova, Product Program Manager, UiPath
UiPath Test Automation using UiPath Test Suite series, part 4DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap.
The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies.
Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques
What will you get from this session?
1. Insights into SAP testing best practices
2. Heatmap utilization for testing
3. Optimization of testing processes
4. Demo
Topics covered:
Execution from the test manager
Orchestrator execution result
Defect reporting
SAP heatmap example with demo
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
Elevating Tactical DDD Patterns Through Object CalisthenicsDorra BARTAGUIZ
After immersing yourself in the blue book and its red counterpart, attending DDD-focused conferences, and applying tactical patterns, you're left with a crucial question: How do I ensure my design is effective? Tactical patterns within Domain-Driven Design (DDD) serve as guiding principles for creating clear and manageable domain models. However, achieving success with these patterns requires additional guidance. Interestingly, we've observed that a set of constraints initially designed for training purposes remarkably aligns with effective pattern implementation, offering a more ‘mechanical’ approach. Let's explore together how Object Calisthenics can elevate the design of your tactical DDD patterns, offering concrete help for those venturing into DDD for the first time!
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Tobias Schneck
As AI technology is pushing into IT I was wondering myself, as an “infrastructure container kubernetes guy”, how get this fancy AI technology get managed from an infrastructure operational view? Is it possible to apply our lovely cloud native principals as well? What benefit’s both technologies could bring to each other?
Let me take this questions and provide you a short journey through existing deployment models and use cases for AI software. On practical examples, we discuss what cloud/on-premise strategy we may need for applying it to our own infrastructure to get it to work from an enterprise perspective. I want to give an overview about infrastructure requirements and technologies, what could be beneficial or limiting your AI use cases in an enterprise environment. An interactive Demo will give you some insides, what approaches I got already working for real.
2. $ whoami
● Fran Garcia
● SRE @hostedgraphite
● “Break fast and move things”
● Absolutely no networking/cryptography background
● No, seriously, totally unqualified to give this talk
3. What this talk is not
A success story
An introduction to IPsec
A HOWTO
A set of best practices
4. What we’ll talk about
Hosted Graphite pre-IPsec
What’s this IPsec thing anyway and why should I care?
Hosted Graphite does IPsec!
Everything we did wrong (well, the least embarrassing bits)
7. In the beginning, there was n2n...
Early days at Hosted Graphite:
- A way to secure communications for riak was needed
- Not many servers to maintain
Enter n2n:
- P2P VPN software
- Supports compression and encryption
- Really easy to setup and maintain
8. Wait, so what’s the catch?
Best description from an HG engineer: “academic abandonware”
Relies on a central node (supernode):
● No supernode, no network
Single-threaded, not really efficient:
● Became a bottleneck, increasing latency for some of our services
Initially configured on a /24 private IP space
● We were running out of IP addresses!
9. Replacing n2n
Our requirements:
- Can’t depend on fancy networking gear
- Cluster spans multiple locations/providers
- We don’t trust the (internal) network!
- Must be efficient enough not to become a bottleneck!
- Simple security model (no complex/dynamic firewall rules)
- Can be implemented reasonably quickly
10. Potential n2n alternatives
We looked at a bunch of possible alternatives and most of them:
- Were not really designed for a full-mesh network (OpenVPN)
- Encrypt data in user space, incurring a performance penalty (tinc)
- Would tie us to a single provider (like AWS VPCs)
- Involve modifying and rearchitecting all our services
- (rolling our own application layer encryption)
So after analyzing all our options IPsec won… almost by default
12. So what’s this IPsec thing anyway?
Not a protocol, but a protocol suite
Open standard, which means lots of options for everything
66 RFCs linked from wikipedia page!
13. What IPsec offers
At the IP layer, it can:
● Encrypt your data (Confidentiality)
● Verify source of received messages (Data-origin authentication)
● Verify integrity of received messages (Data Integrity)
Offers your choice of everything to achieve this
14. Choices, choices everywhere
What protocol?
● Authentication Header (AH): Just data integrity/authentication*
● Encapsulating Security Payload (ESP): Encryption + integrity/auth (optional)
● AH/ESP
(TL;DR - You probably want ESP)
*Legend says AH only exists to annoy Microsoft
15. Second choice... Tunnel or Transport mode?
*Transport mode might incur in a slightly smaller overhead and be a bit simpler to set up
Encapsulates
header
Encapsulates
payload
Works for host-
to-host
Works for site-
to-site
Tunnel Mode YES YES YES YES
Transport Mode NO YES YES NO
16. IPsec: What’s a SP (security policy)?
Consulted by the kernel when processing traffic (inbound and outbound)
“From host A to host B use ESP in transport mode”
“From host C to host D’s port 443 do not use IPsec at all”
Stored in the SPD (Security Policy Database) inside the kernel
17. IPsec: What’s a SA (Security Association)?
Secured unidirectional connection between peers:
- So need two for bidirectional communication (hosta->hostb, hostb->hosta)
Contains keys and other attributes like its lifetime, IP address of peer...
Stored in the SAD (Security Association Database) inside the kernel
18. IKE? Who’s IKE?
“Internet Key Exchange”
Negotiate algorithms/keys needed to establish secure channel between peers
A key management daemon does it in user space, consists of 2 phases
19. IPsec: IKE Phase 1
Lives inside the key management daemon (in user space)
Hosts negotiate proposals on how to authenticate and secure the channel
Negotiated session keys used to establish actual (multiple) IPsec SAs later
20. IPsec: Phase 2
Negotiates IPsec SA parameters (protected by IKE SA) using phase 1 keys
Establishes the actual IPsec SA (and stores it in SADB)
Can renegotiate when close to end of lifetime
21. Life of an IPsec packet
Packet
arrives
Check
SPD
Carry on
Is there a
existing
SA with
this host?
Use it!
Kernel notifies key
management daemon via
PF_KEY(RFC 2367) to
establish SA
IPsec not required
IPsec
required
YES
NO
SA established
Kernel
Userspace
22. Some helpful commands
ip xfrm is pretty powerful. Some basics:
$ ip xfrm policy # Dump the contents of the SPDB
$ ip xfrm state # Dump the contents of the SADB
$ ip xfrm monitor # Dump all changes to SADB and SPDB as they happen
$ ip xfrm state flush # Flush all state in the SADB (dangerous!)
Documentation is... not great: http://man7.org/linux/man-pages/man8/ip-xfrm.8.html
23. So what has IPsec ever done for us?
Encryption happens inside the kernel, so it’s fast!
Using the right algorithms/settings it can be fairly secure
It’s a standard, so there are good practices to use it securely
Very flexible, which is useful if you have:
- Hardware distributed across different datacenters/providers
- No real control over your network infrastructure
25. Our migration: n2n -> IPsec
Big time constraints: n2n was unreliable and preventing us from scaling
We had trouble finding reports of people using IPsec in the same way*…
...So we had to improvise a bit.
After careful planning and testing we rolled it out to our production cluster...
* Notable exception: pagerduty’s Doug Barth at Velocity 2015 http://conferences.oreilly.
com/velocity/devops-web-performance-2015/public/schedule/detail/41454
29. WORST. MIGRATION. EVER
Migration attempt resulted in multi-day incident:
http://status.hostedgraphite.com/incidents/gw2v1rhm8p5g
Took two days to stabilize, a full week to resolve the incident.
Lots of issues not found during testing
30. n2n -> IPsec migration aftermath
Back to drawing board, came up with another plan
Spent almost 3 months slowly rolling it out and fixing bugs:
- Also known as “the worst three months of my life”
- Big team effort, everybody pitched in
Still worth it, things are stable now and we’ve learned a lot
32. Our IPsec stack: present day
Hundreds of hosts using ESP in transport mode (full-mesh)
Several clusters, isolated from each other
Using ipsec-tools with racoon as key management daemon
33. Our config: iptables
# Accept all IKE traffic, also allowing NAT Traversal (UDP 4500)
-A ufw-user-input -p udp --dport 500 -j ACCEPT
-A ufw-user-input -p udp --dport 4500 -j ACCEPT
# Allow all ESP traffic, if it has a formed IPsec SA we trust it
-A ufw-user-input -p esp -j ACCEPT
34. Our config: Security Policies (/etc/ipsec-tools.conf)
Node1 = 1.2.3.4 Node2 = 5.6.7.8
On node1:
On node2:
# require use of IPsec for all other traffic with node2
spdadd 1.2.3.4 5.6.7.8 any -P out ipsec esp/transport//require;
spdadd 5.6.7.8 1.2.3.4 any -P in ipsec esp/transport//require;
# require use of IPsec for all other traffic with node1
spdadd 5.6.7.8 1.2.3.4 any -P out ipsec esp/transport//require;
spdadd 1.2.3.4 5.6.7.8 any -P in ipsec esp/transport//require;
35. Our config: Security Policies (/etc/ipsec-tools.conf)
What about management hosts?
Node1 = 1.2.3.4 PuppetMaster = 5.6.7.8
On node1:
Everything else will get dropped by the firewall
# Only require IPsec for port 8140 on the puppet master
spdadd 1.2.3.4 5.6.7.8[8140] any -P out ipsec esp/transport//require;
spdadd 5.6.7.8[8140] 1.2.3.4 any -P in ipsec esp/transport//require;
36. Our config: Security Policies (/etc/ipsec-tools.conf)
# Exclude ssh traffic:
spdadd 0.0.0.0/0[22] 0.0.0.0/0 tcp -P in prio def +100 none;
spdadd 0.0.0.0/0[22] 0.0.0.0/0 tcp -P out prio def +100 none;
spdadd 0.0.0.0/0 0.0.0.0/0[22] tcp -P in prio def +100 none;
spdadd 0.0.0.0/0 0.0.0.0/0[22] tcp -P out prio def +100 none;
# Exclude ICMP traffic (decouple ping and the like from IPsec):
spdadd 0.0.0.0/0 0.0.0.0/0 icmp -P out prio def +100 none;
spdadd 0.0.0.0/0 0.0.0.0/0 icmp -P in prio def +100 none;
39. 10 DOS AND
500 DONT’S Disclaimer:
(We don’t really have 10 dos)
40. Don’t use ipsec-tools/racoon! (like we did)
Not actively maintained (Last release on early 2014)
Buggy
But the only thing that worked for us under time/resource constraints
LibreSwan seems like a strong alternative
41. “The mystery of the disappearing SAs”
Some hosts unable to establish SAs on certain cases
Racoon would complain of SAs not existing (kernel would disagree):
ERROR: no policy found: id:281009.
racoon’s internal view of the SADB would get out of sync with the kernel’s
We suspect corruption in racoon’s internal state for the SADB
42. “The mystery of the disappearing SAs”
Restarting racoon fixes it, but that wipes out all your SAs!
Workaround: Force racoon to reload both SADB and config
killall -HUP racoon
Forcing periodic reloads prevents the issue from reoccurring ¯_(ツ)_/¯
43. Don’t blindly force all traffic to go through IPsec
Account for everything that needs an exception:
- SSH, ICMP, etc
You’ll need to be able to answer these two questions:
- “Is the network broken?”
- “Is IPsec broken?”
44. “Yo dawg, I heard you like encrypted traffic…”
If migrating from an existing VPN, make sure to exclude it from IPsec traffic
During our initial rollout our SPs forced our n2n traffic through IPsec…
… Which still wasn’t working reliably enough…
… Effectively killing our whole internal network
45. Don’t just enable DPD… without testing
What’s DPD?
● DPD: Dead Peer Detection (RFC3706)
● Liveness checks on Phase 1 relationships
● If no response to R-U-THERE clears phase 1 and 2 relationships…
Sounds useful but test it in your environment first:
● racoon implementation is buggy!
46. “The trouble with DPDs”
In our case, enabling DPD results in 100s of SAs between two hosts:
- Every failed DPD check resulting in extra SAs
Combination of factors:
- Unreliable network
- Bugs in racoon
We ended up giving up on DPD
47. Don’t just disable DPD either
DPD can be legitimately useful
Example: What happens when rebooting a host?
Other nodes might not realise their SAs are no longer valid!
48. DPD: Rebooting hosts
bender’s SAD:
5.6.7.8 -> 1.2.3.4 (spi: 0x01)
1.2.3.4 -> 5.6.7.8 (spi: 0x02)
flexo’s SAD:
5.6.7.8 -> 1.2.3.4 (spi: 0x01)
1.2.3.4 -> 5.6.7.8 (spi: 0x02)
These are two happy hosts right now…
bender -> flexo (using spi 0x02) traffic is received by flexo
flexo -> bender (using spi 0x01) traffic is received by bender!
… But let’s say we reboot bender!
49. DPD: Rebooting hosts
bender’s SAD:
5.6.7.8 -> 1.2.3.4 (spi: 0x02)
1.2.3.4 -> 5.6.7.8 (spi: 0x01)
flexo’s SAD:
5.6.7.8 -> 1.2.3.4 (spi: 0x02)
1.2.3.4 -> 5.6.7.8 (spi: 0x01)
bender’s SADB is now empty
flexo->bender traffic (using spi 0x02) will be broken until:
● bender->flexo traffic forces establishment of new SAs
● The SAs on flexo’s side expire
50. DPD: Just roll your own
Our solution: Implement our own phase 2 liveness check
Check a known port for every host we have a mature SA with:
- Clear the SAs if ${max_tries} timeouts
Bonus points: Also check a port that won’t use IPsec to compare
51. Do instrument all the things!
You’ll ask yourself “is the network broken or just IPsec?” a lot
So better have lots of data!
Built racoon to emit timing info on logs (build with --enable-stats)
Diamond collector gathers and send metrics from:
- racoon logs
- SADB
54. Do instrument all the things!
Kernel metrics also useful (if available!)
You want your kernel compiled with CONFIG_XFRM_STATISTICS
$ cat /proc/net/xfrm_stat
XfrmInError 0
XfrmInBufferError 0
…
(Very) brief descriptions: https://www.kernel.org/doc/Documentation/networking/xfrm_proc.txt
55. Instrumenting kernel xfrm stats: XfrmInNoStates
Wouldn’t want to be
on call here!
XfrmInNoStates: Times we’ve received data for an SA we know nothing about
56. “The case of the sad halfling”
bender’s SAD:
5.6.7.8 -> 1.2.3.4 (spi: 0x02)
1.2.3.4 -> 5.6.7.8 (spi: 0x01)
flexo’s SAD:
5.6.7.8 -> 1.2.3.4 (spi: 0x02)
1.2.3.4 -> 5.6.7.8 (spi: 0x01)
These are two happy hosts right now...
… But let’s say one SA “disappears” during a brief netsplit:
bender$ echo “deleteall 5.6.7.8 1.2.3.4 esp ; ” | setkey -c
# The 5.6.7.8 1.2.3.4 association gets removed from bender
57. “The case of the sad halfling”
bender’s SAD:
5.6.7.8 -> 1.2.3.4 (spi: 0x02)
1.2.3.4 -> 5.6.7.8 (spi: 0x01)
flexo’s SAD:
5.6.7.8 -> 1.2.3.4 (spi: 0x02)
1.2.3.4 -> 5.6.7.8 (spi: 0x01)
Any communication attempt will fail!
bender -> flexo (using spi 0x01) traffic is received by flexo
flexo -> bender (using spi 0x02) traffic is ignored by bender!
58. “The case of the sad halfling”
We built a custom daemon for detecting it
Highlights need for:
- Phase 2 liveness checks
- Metrics for everything!
59. Don’t flush/restart on changes
Never restart racoon!
A racoon restart will flush all phase 1 and 2 SAs:
● Negotiating ~1000 SAs at once is no fun
To flush an individual SA: ip xfrm state delete
To reload config changes: killall -HUP racoon
60. Don’t flush/restart on changes
When adding/removing hosts, do not flush both SPD and SAD!
Just flush/reload your SPD and let unwanted SAs expire
SAs not reflected in the SPD will never get used
Can flush that SA individually if feeling paranoid
Can just include spdflush; in your ipsec-tools.conf and reload with:
setkey -f /etc/ipsec-tools.conf
61. You don’t have the same tools available
tcpdump will just show ESP traffic, not its content:
15:47:51.511135 IP 1.2.3.4 > 5.6.7.8: ESP(spi=0x00fb0c52,seq=0x1afa), length 84
15:47:51.511295 IP 5.6.7.8 > 1.2.3.4: ESP(spi=0x095e5523,seq=0x173a), length 84
Traffic can be decrypted with wireshark/tshark if you dump the keys first
62. You don’t have the same tools available
Can use tcpdump with netfilter logging framework:
$ iptables -t mangle -I PREROUTING -m policy --pol ipsec --dir in -j NFLOG --nflog-group 5
$ iptables -t mangle -I POSTROUTING -m policy --pol ipsec --dir out -j NFLOG --nflog-group 5
$ tcpdump -i nflog:5
Doesn’t allow most filters
Might need to increase the buffer size
63. You don’t have the same tools available
Traceroute will attempt to use udp by default:
You can force it to use ICMP with traceroute -I
$ traceroute that.other.host
traceroute to that.other.host (5.6.7.8), 30 hops max, 60 byte packets
1 * * *
2 * * *
3 * * *
4 that.other.host (5.6.7.8) 0.351 ms 0.295 ms 0.297 ms
64. Do use certs for auth, or don’t use a weak PSK
PSK is great for getting started if you don’t have PKI in place (we didn’t)
But please:
● Use a strong PSK (if you must use PSK)
● Enable PFS (Perfect Forward Secrecy)
● Do not use aggressive mode for phase 1
Not following all that makes the NSA happy!
65. Don’t trust the (kernel) defaults!
Careful with net.ipv4.xfrm4_gc_thresh
Associations might be garbage collected before they can succeed!
If 3.6 > $(uname -r) < 3.13:
Default (1024) might be too low
GC will cause performance issues
Can refuse new allocations if you hit (1024 * 2) dst entries
66. Don’t trust the defaults!
Beware of IPsec overhead and MTU/MSS:
A 1450 bytes IP packet becomes:
● 1516 bytes in Transport mode
● 1532 bytes in Tunnel mode
(More if enabling NAT Traversal!)
Path MTU Discovery should help, but test it first!
67. Thanks!
Hated it? Want to say hi?
fran@hostedgraphite.com
@hostedgraphite
hostedgraphite.com/jobs
Questions?