How we stopped a DDos ransom attack - some redaction in deck - why give opportunists to up their game?

1. ASK ME WHY
I DON'T LIKE DDOSERS
AND I’LL TELL YOU WHY
YOU SHOULD NEVER PAY THE RANSOM
https://faelix.link/netmcr59

2. About Marek
Stuff I do:
CTO @FAELIX – https://faelix.net/
PC @uknof – https://uknof.uk/
Crew @net_mcr – https://www.netmcr.uk/
Trail of SSIDs in my wake: "AS41495 Faelix Limited"
Me — @maznu – @NetworkMoose

3. These Slides are
Not all slides are included in this public deck
Is this “security through obscurity”?
Not giving opportunists ideas to up their game

4. Customer Received This…
To stop these attacks, please deposit in bitcoin
to $address before

5. <RECORD SCRATCH>
WHAT ABOUT LAST WEEK?

6. Symptoms
Call from occasional-consultancy client saying:
Connectivity problems
Difficulty logging onto the routers at each site
Struggling to diagnose
Please help!?
External observations:
BGP sessions on LON1 were down/flapping
One of their customers also alerted us to problems

7. bad stuff
and more
bad stuff

8. UDP 53+389; some ICMP

9. Sudden Start, Sudden Stop

10. Initial Thoughts
Looks like a DDoS
Feels like a DDoS
Was not preceded by a ransom note…?
One-off attack?
Are any access customers gamers?
Did a ransom email go missing?
What was targeted?

12. DDoS Traffic from IX
Customer’s ASN peers with LINX route-servers
And some networks who sent peering requests
Not every peering request is worth your time
If control plane is weak, decide with netflow stats
Also consider whether a peer might be a path for
e.g. compromised eyeballs to send you 1Mpps

13. Weak Control Plane
DDoS pegged your CPU? BGP is going to hurt:
Router in attack path cannot keep up BGP hellos
eBGP sessions (with shorter timers) drop
Huge route churn to iBGP sessions
Even more CPU load, might drop iBGP
Cascades across entire iBGP mesh
Vicious cycle of instability

14. Targets

15. Initial Steps
Accept default + full tables from upstreams
Filter full tables and only accept “golden networks”
Filter IXs (especially LINX RS) for “golden networks”
Pare down from approx 2x 860k and 2x 200k routes
To just ~210k total (~105k active)
Convergence reduced to ~1 minute (from 10+)

16. Next Steps
Still no ransom note!
Is this attacking a specific customer(s)?
Implement some quick fixes:
Inject /32 route to BGP P+T routers
Blackhole traffic on entry to network

17. Strengths / Luck
Very obvious pattern of the attacks now

18. Challenges / Suck
Lots of UDP with no specific ports

19. Attacks Move
Next attack targets
Why is this happening? Still no ransom note!
Evolve defences:
Send flow data to a VPS
Run fastnetmon to detect badness and inject /32
Builds on the manual approach we had in place

20. fastnetmon Difficulties
Attacks are bit/sec at each of IPs
Number of PPS? Mbit/sec? Flows?
From experience: false-positives block customers
We needed several iterations of tuning fastnetmon
Most recent was ~4 weeks after implementation!

21. Balancing Act
Easy for a human to spot an attack
Not so easy for a “simple” tool like fastnetmon
Absolutely need to tune the config!
Initial “manual approach” doesn’t react quickly
But “manual approach” less likely to false positive
Once we understood more about the attacks, we
moved to auto-blocking

22. fastnetmon_cli

23. Ransom Note Arrives

24. My Thoughts on Ransoms
Badness will continue until BTC account improves
Increasingly “cyber insurance” is paying ransoms
Is there honour among thieves?
Or are you now known to be a soft target?
Gives the crooks more resources to spend on
stresser or DDoS tools for hire
Extends the pain to other networks
Crime should not pay!

25. Get a Scrubber
AS41495 has Voxility as an upstream transit
We buy DDoS mitigation with that service
We can apply that to downstream networks
Add Victim ISP to AS-FAELIX
Order (and expedite) XC
Establish direct peering over LINX
Propagate their routes to AS3223

26. <WOBBLY DISSOLVE
TRAVEL BACK IN TIME>

27. UKNOF September 2020

28. Whoops!
AS41495 applies BCP38 to peering and transit ports
Will drop packets trying to go out a P/T port if source
address is not within AS-FAELIX
This will drop packets that have come from outside
AS-FAELIX destined for Victim ISP
We need to provide Victim ISP transit over LON1
Therefore we will need to emit packets from
0.0.0.0/0 out of AS4145’s LON1 port

29. Sorry, Caida!

30. Sorry, EveryoneJISC!
LINX LON1

31. Nearing the Deadline
Another attack happens
fastnetmon does its thing
Voxility does their thing
AS41495’s LON1 port was warm for a minute
Was still a bit of pain (still tweaking knobs) but feels
like we’re getting a hold of the situation

32. Finally: Cross-Connect!
Add Victim ISP as a “traditional” downstream
Adjust BGP session on LON1 to be normal peering
Victim is still protected… deadline is hours away!
AS41495 will re-apply BCP38 filtering on LON1 port
a few days later

33. The Deadline
…?
…?
Seriously…?

34. Oh! There It Is!

35. Oh! There It Is!
???

36. Even Scrubbers Overblock

37. Contact from Ransomers
Ransom:
Follow-up:

38. But Wait! There’s More!

39. “We will not go away
until you pay.”
– paraphrased from ransom note

40. NARRATOR:
THEY DID GO AWAY

41. Why Did Attacks Stop?
Combination of mitigations were effective:
Their ASN is no longer a soft target
Attackers moved on to other low-hanging fruit

42. Anecdata

43. Anecdata
Help your upstreams by using their RTBH
Some tier-1s have selective RTBH communities
e.g. drop only from country/region/ASN

44. What Does It Feel Like?
For Victim ISP: pain, “some of the most stressful
days” their head of network has ever had
For Victim ISP’s customers: unpredictable pain
For Incident Responders: try to remain cool and
methodical, analyse and iterate, learn and improve
defences, plan for next time

45. What We Learned
Don’t be complacent just because no ransom note
Even with expedite, XCs can take (what feels like)
forever to get installed
Transit over LINX LON1 vs BCP38
DDoS mitigation is expensive, at least till you need it
Traffic engineering BGP communities are cool

46. RECOMMENDATIONS

47. Some Ideas
Be able to blackhole routes across entire network
Be ready to drop ingress traffic
But don’t you dare block all ICMP!

48. Next Steps
Get a real out-of-band network to all your POPs
Get RTBH from the upstreams that support it
Understand your flows from customers
Understand traffic patterns with peers/upstreams
Deploy tooling, start tuning it before you need it
Join Team Cymru’s UTRS
“Who ya gonna call?” when it gets too big

49. ASK ME HOW
MANY MORE
GRAY HAIRS
E: marek @ faelix . net
T: @maznu
T: @faelix
W: https://faelix.net/
https://faelix.link/netmcr59