copyright IOActive, Inc. 2006, all rights
2008 and the new (old) nature of
Director of Penetration Testing
What a year!
• Significant flaw found in DNS
– You might have heard about it
• This talk is not going to be a repeat of the
DNS hacking talk
– If you want that, the video and slides are
– Or you’re welcome to ask me
• This talk has a somewhat different focus
• 1) Quickly recap what the flaw was, and the scope
of things it affected
• 2) Analyze what forces allowed what should have
been a non-security-sensitive technology to break
so many things
• 3) Recount how we managed to conspire to get
the issue fixed
• 4) Discuss DNSSEC, and why it may not be such
a bad idea after all
The Flaw (1999 Edition)
• 1999: DJB says 16 bit transaction ID’s on queries
aren’t enough – attacker can brute force and
– DNS community responds: “There has to be a
query waiting for a response, for an attacker to
guess a response. The TTL – Time To Live –
limits how rapidly an attacker can force new
queries awaiting responses. So if the TTL is
one day, an attack will take years!
• This almost became an RFC – “Forgery
Resilience” – advocating long TTL’s
The Flaw (2008 Edition)
• 2008: I point out that there are many, many ways to get
around the TTL defense
– Really, that’s it.
• Maybe I also found that since the attacker controls
when the query occurs, he can reliably get hundreds
of replies in before the real reply arrives
– Without the TTL slowing down the attack, the attack
– The defense against DJB’s attack didn’t work
• But then, it was 1999, most security in 1999 didn’t
• What should have happened
– No important systems should have been
• “I fail to understand the seriousness with
which this bug is handled though. Anybody
who uses the Internet has to assume that his
gateway is owned.”
• What actually happened
– Anybody != Halvar
Not Repeating All The Slides, But…
• “Secure” systems are actually pretty rare in the field
– Most things don’t even bother
• Vast majority of the web
• Non-browser network applications
– Those that try, mostly fail
• 41% of SSL certs are self-signed
– “Who are you encrypting to?” “I DON’T KNOW!”
• Non-browser network applications that use SSL tend not to
care if the cert is signed by anyone
– There are some pretty scary implications
• Automatic updaters are non-browser network applications
that assume DNS is safe
• SSL certificates depend on email to authenticate receivers
• “Forgot My Password” systems bypass auth entirely
– I don’t think people understand how serious that is
What Just Happened?
• We can forget our passwords, and have them mailed to us.
– Admins have passwords too.
– Admins have code execution rights on pretty much every CMS
• Not just picking on Drupal here!
– Working closely with them on building a test module in
– this isn’t a bug in their code, any more than a
vulnerable TCP stack might be
• You think this wouldn’t work on almost every other real
• We just received a code-execution equivalent token over email
• “I fail to understand the seriousness with which this bug is
handled though. Anybody who uses the Internet has to
assume that his gateway is owned.”
• Why did this work?
– Ah, thus the subject of this talk.
Obviously, this is the fault of
• Without passwords, there would have been nothing to forget
• With nothing to forget, there would have been no need for a
• Without email, there would have been no dependency on
• Without DNS, there would have been no exposure to cache
• So clearly, we need to stop using passwords and only use
SSL client certificates!
– Strong crypto
– Global PKI
– $10 per user
• What, no rush to sign up?
If You Don’t Understand The Problem,
You Won’t Find The Solution
• We have some bad habits in the security industry:
– Appeal to Morality: “If you were a good programmer,
you’d use SSL Client Certificates instead of passwords.”
– Appeal to Intelligence: “If you were a smart programmer,
you’d use SSL Client Certificates instead of passwords.”
– This is nice because it makes us feel good and smart
• Reality: What can take a team of programmers two
years to build, can take us two weeks to break.
– We have the easy job.
– Neither argument is very compelling.
• SSL client certificates are awkward and expensive to
manage, and fail many critical use case scenarios
– Put another way: PASSWORDS SCALE
• So does DNS – like nothing else does.
Federation Is Hard.
• Definition of Federation: the formation of a political unity, with a central
government, by a number of separate states, each of which retains
control of its own internal affairs.
– Put another way: Microsoft doesn’t trust Google. Google doesn’t
trust Yahoo. Yahoo doesn’t trust CNN. All share however a single
namespace (the DNS), all control operations within their
• Federation is a hard problem
– Requires technology
• Synchronization of distributed databases is a classically hard
– Requires more than just technology
• Managing who is trusted to update what record there is as
much a human problem, as it is a technical problem
• DNS had first mover advantage, being built in 1983
– Every IT shop has someone whose job it is to update the DNS
– Interactions with the global DNS are limited after initial registration
Everyone Federates With DNS
– To send a mail, check DNS to determine which server to initiate
• There’s even a special record type -- MX
• The Web
– “Same Origin Policy”
• Arguably the largest advance in security technology in the last
– To determine whether one entity can access another, compare their
– Supposedly the real federated network
• Not very reliably federated: Which root CA’s do you or do you
• Not very federated: Wildcard certs are difficult to acquire and
unreliable, so constant cross-company interaction required
• Not actually independent of DNS
Everyone federates with DNS
• Password resets use email, so that
passwords only go to the user who owns
• OpenID uses the web and its Same Origin
Policy, so that different sites can use the
same authentication server safely
• SSL uses email, so that only the user that
controls a domain can acquire a signed
certificate for it
But There’s A Problem
• DNS tells you how to get there, but it doesn’t tell
you what to expect when you arrive.
– It’s the worldwide, distributed, fully federated
database that reasonably secures everything
going into the database…but can’t validate
anything coming back out.
• Public Key Infrastructure…without the keys
• Theory: Because DNS doesn’t secure its content,
nobody will treat its payloads as security critical
• Reality: It’s the only thing that can scalably tell
you where to go. People are using it anyway.
• DNS tells you where to go, but not who to expect when you arrive.
• Email imports DNS. Email knows where to go, but not who not to
deliver mail to.
• The web (HTTP) imports DNS. The web knows where to go, but
not if an ISP has changed anything.
• Password resets import email, which imports DNS, know where to
go, but not actually who they’re being delivered to.
• DNS’s inability to authenticate replies surfaces as a failure to
authenticate in system after system after system
– We can deny these systems exist
– We can insult their authors
– We can pat ourselves on the back
– Or we can start dealing with our inability to authenticate.
2008 Has Been A Rough Year For
• Mike Zusman: SSL VPN’s, which have no other purpose but to
prevent bad guys from intercepting traffic, don’t make sure a bad
guy isn’t intercepting traffic
– They aren’t validating SSL certs
• Mike Perry: HTTP Cookies received over SSL, which are used to
authenticate access to https://www.bank.com, are by default also
sent to http://www.bank.com. Most sites do not change the default.
• Francisco Amato’s Evilgrade: You’d think anyone who went out
onto the Internet and said, “Dear Internet, please give me arbitrary
code to execute”, would actually authenticate the code retrieved.
But in general, they don’t.
– Java plugin - Winzip - Winamp - MacOS - OpenOffices -
iTunes - Linkedin Toolbar - DAP [Download Accelerator] -
notepad++ - speedbit
Bug Recap: Luciano Bello’s Debian
Random Number Generator
• To authenticate a foreign host, we ask it to:
– 1) Generate a public and private key
– 2) Get the public key signed by a mutually trusted third party
– 3) Prove possession of the private key matching the signed
• Math says, can’t determine the private key from the public
• But what if the private key isn’t random?
– Then anyone who can see the public key, can figure out the
• Debian broke their random number generator
• So we accepted a bad guy’s key exchange as legitimate, because
the bad guy could discover the private key
Bug Recap: Wes Hardakar’s SNMPv3
• Probably the coolest bug of the year
• SNMP: Simple Network Management Protocol
– Used to control access to core network infrastructure
– Access from Internet required for LE support purposes
• To authenticate access to routers, a challenge is sent, and the client
needs to hash the correct password with the challenge
– Only if the hash sent is correct, will access be granted
– But what does it mean to be “correct”?
• The entire hash sent by the client must be correct
– What if the client could specify only the first byte needed to be
• There are only 256 possible bytes…
• The router would accept authentication from anyone who
tried all 256
Common Trait #1:
Flawed Authentication Is The Unifying
Theme Of 2008’s Major Bugs
• From SSL to Cookie Monster, from SNMPv3 to
Evilgrade, we just don’t know who we’re talking to
and it’s getting us in trouble
– Weaknesses in authentication and
encryption, some which have been known
to at least some degree for quite some time
and many of which are sourced in the core
design of the system, continue to pose a
threat to the Internet infrastructure at large,
both by corrupting routing, and making
those corrupted routes problematic.
Common Trait #2: It’s All So Simple
• DNS’s TTL defense can be bypassed using
• Auth Cookies received securely will be sent
• SSL-VPN’s don’t care who they deliver updates to
• Large amounts of software doesn’t care who they
get their updates from
• These are not complicated bugs to understand
Common Trait #3: It’s All So
• Quote from a release manager at a company that has to
manage large amounts of public security research
– “I love buffer overflows!”
• Nobody depends on them
• No need to migrate people away from them
• (Usually) no need to collaborate with other companies
that suffer from them
– Rats nest of multivendor bugs
• (Usually) a small amount of code that needs to be
– Sometimes even a small number of versions that
• Design bugs aren’t like that
– “Regression tested for vulnerabilities”
Common Trait #4: They Blend
• DNS and SNMPv3 let you become Man-In-The-Middle
– Receive authentication cookies from Cookie Monster
– Impersonate SSL endpoints secured by Debian
– Provide false packages via EvilGrade
• DNS lets you exploit SNMPv3
– DNS used to control access to Java’s socket libraries
• Bad DNS = Java applets can speak UDP to any IP address
• SNMPv3 runs over UDP
• So if you can corrupt DNS, you can corrupt internal routers
• Consequence of simplicity
– This is why it’s ridiculous to compare “who has the bigger bug”:
There is no bug so good that another bug won’t make it better
Common Trait #5: Often Found In
• DNS bug was in a spec from 1983
• Cookie spec from mid-90’s
• SSL requirements (cert checking, random
keying) have been known forever
• SNMPv3 spec has been around for years
• Age does not predict quality
– We didn’t used to care about security
Common Trait #6: Very, Very Slow
and Expensive To Repair
• DNS fix probably the largest simultaneous patching
operation of all time
– Was an amazing amount of work just to minimize the
disruption from the fix
– Two days to find the bug, eight months to fix it
• And it was a pretty simple fix! Nobody depended on it
to be broken!
– Paul Vixie’s expected success rate: 50% after a year
– Actual success rate: ~75% after a month
• Major sites still struggling to adapt to Cookie Monster attacks
• SSL-VPN’s still widely broken despite public knowledge of
An Interesting Theory
• Historical games: “What if?”
– What if DNS had been secure from the start?
– Would all of these design bugs have actually
• SNMPv3 and NRNG definitely would have
• But what of SSL-VPNs, Cookie Monster, and
Why are SSL-VPN’s insecure?
• My original reaction to hearing that SSL-VPN’s didn’t authenticate who
they were encrypting to was shock and indignation
– Appeal to Morality and Intelligence
– Blamed “commercial realities”
• But then I started listening
– “We have to ship test gear to customers. It has to still work after
they buy it. How are customers supposed to get their certificates?”
• The present system for securing SSL, for devices, requires interactions
between three companies – a device manufacturer, a customer, and a
third party root CA
– Inter-company interactions are expensive
– This is completely inelegant
– And yet, every one of those devices is showing up in DNS.
• DNS scales. The SSL infrastructure, not so much here.
• A question: If DNS had been a secure place to put certificate hashes,
might SSL VPN’s have just told people “heh, throw this in your DNS w/
Why are HTTP Cookies via SSL
• There is no good way to know that a site is
– So it’s common that we try to access things
insecurely, and then upgrade
• 302 redirect to HTTP
– Which can of course be hijacked
– Also common that we need to be logged into
both the secure and insecure side of a site
• What if we had a secure, scalable way to declare
that a site was HTTPS only?
Why are automatic upgrades
• An automatic upgrade over SSL would have almost every
security property of Windows Update
– Missing property: Signing box offline in a vault
– It would also be much slower, since SSL session negotiation
is much slower than TCP session negotiation
• It Has To Work
• People distribute updates over HTTP because it scales
– It’s very difficult to correctly generate signatures across
packages and validate them
• Wrong version
• Wrong package
• Wrong signing key
• What if there was a secure, scalable way to distribute hashes
for new versions of packages?
• What if we had a secure, scalable place to
put PGP keys?
– Maybe we’d finally have secure email.
Commercial Realities Are A Crutch
• Have we been blaming the business guys for what’s
ultimately just poor engineering?
• The systems we are trying to build, to make up for the fact
that DNS is insecure, are resource intensive and just do not
• We’ve spent the last year finding design bugs that break
– Maybe there’s something fundamentally missing, that
keeps forcing these bugs in
• Perhaps DNS shouldn’t be at the heart of authentication.
But it is, and it’s time we start treating it that way.
The “Shouldn’t Be In DNS” Problem
• There are some who think that people “shouldn’t put strange
things in DNS”
– “It wasn’t designed for that”
• Well, the jig is up: A whole mess of things have been seen in
– SSH Fingerprints (SSHFP – how is that secure?)
– Anti-SPAM TXT records
– VoIP directories
– IPSec keys (again, how is that secure?)
• The reality of federation: Nobody gets to tell you what you
host in your own DNS
– Whether you cache or not is your choice, but that’s it
• We’re using it for auth. We might as well do it securely.
So what’s it going to take?
• First, put out the immediate fire
– What we just did
• Next, figure out how to make DNSSEC
– It doesn’t yet
• Finally, start migrating new applications to it
– This adds its own layer of difficulty
Putting Out The Fire: How We Did This? (Animation,
Joachim Viide et al, Clarified Networks)
How We Did This
• 1) Identified the critical players.
– Paul Vixie did an amazing job here, with absolutely no care in the
world for politics. This would not have worked without Paul.
• Paul pulled in critical engineers from across the industry.
• 2) Met in person
– Email is good for many things.
– Email is terrible at forcing decisions to be made
• An infinite number of factions can break off, an infinite number
of people can join in
– This is not good for keeping secrets or fixing stuff
– Paul brought in engineers with decision making capacity
• This is how we kept the politics out
• 3) Agreed to synchronize our fixes
– The bug class – that TTL’s can be bypassed – was simple enough
that if any went public, it would harm everyone else’s customers
So what were (and are) the ground
rules for any fix?
• 1) We must secure all names, not just “most”
– Any “missing names” are next year’s Black Hat talk
• 2) We must secure all authoritative name servers, not just those
that opt in
– This eventually gets relaxed with DNSSEC, but that’s not a
solution for today
– This does mean that we must secure the link to the root
servers, and the com servers, without actually requiring root or
com to update anything
• If root or com is busted, then we never get routed to our
“secure” name servers
• 3) We must not alter the semantics of DNS
– If 1% of names stop resolving, people will roll back the patch
The “Easiest” Fix
• DJB was right.
– Source Port Randomization (SPR) is not elegant.
• It’s slow.
• It’s difficult when there are other services on the host.
– Port conflict.
• It doesn’t play well with firewalls, which were way more
prevalent than we expected.
– It meets the ground perfectly, though:
• Secures all names equally – there’s no sort of name that
doesn’t have a port or TXID attached to it
• Secures all name servers – the com and root servers must
support UDP Source Ports, by simple TCP/IP semantics
• Does not alter the semantics of DNS – whatever would
have been returned by DNS before, will still be returned
– It’s also very very simple
• Every other fix is a melange
Do not underestimate the value of
• We knew there were deeply complicated
DNS fixes possible.
• We also knew that not only would they not
be comprehensive, but the odds of all
parties creating interoperable and
compatible implementations of the weirder
stuff were very low
• “Can we bring back the TTL?”
– With at least 15, and possibly dozens of variants on TTL
bypass, there’s just no way to be comprehensive
– Here’s a fun one I just learned about, care of Florian Weimer
• Query for unknown type (TYPE1337)
• Flood with replies that say NXDOMAIN
• By DNS semantics, must now remove all records attached
to that domain, independent of type
• So, after you lose a race, you can clear the slate
– This approach ends up becoming useful defense in depth, but is
• You end up breaking semantics long before you make
things meaningfully more secure
• All “cache scoping” fixes end up being a retreat to TTL
• May have a role in a “midpoint fix” though
Other Tools At Our Disposal
• TCP: This is apparently deeply non-performant, i.e. the DNS
would fall over and die.
• Attack Mode: Deploy defenses only when being attacked at
a certain rate
– Activates defenses only when there’s a confirmed
• Attacking post-SPR is many things, but subtle is not
one of them
– Billions of packets!
– But what to do?
• Can’t block entirely – DNS is spoofable, this becomes
a trivial DoS
• Double/Triple Resolve
– Multiple queries go out with the same request, with
different entropy. Overlap between responses is treated
as the returned data
• Good for attack mode
• Bad for CDN’s (Akamai/Limelight) – “volatile records”
• 0x20 (David Dagon’s trick)
– DNS is case insensitive but case preserving –
wWw.gOOgLE.CoM will resolve, and the response will
retain the case
– Those are useful bits of information – 1 bit per character
– Fails on names like a1.111111111111
• No uppercase 1
Should there be a midpoint fix?
• I don’t know.
• Nicholas Weaver from UC Berkeley is doing some
excellent work to enumerate possibilities.
• Any post-SPR fix would be far more complicated
and fragile than SPR itself was.
– We knew, then, to leave the post-SPR work to
after the announce.
What of the release?
• Most of what happened after July 8th
– Two interesting things I’d like to go into
more depth on, however
• A) Notifications
• B) The testing app
• How many people in this room have been
– …steal credit card numbers?
– …run off with a laptop containing gigs of
– …extract data from the servers of a
• Now how many have been
asked to break into someone’s
Hotmail, or GMail, or Yahoo?
• For sheer chaos reduction,
quite a bit of energy was spent
bringing the “cloud” up to
speed regarding the server
implications of this flaw
– Citrix (GoToMyPC)
• We also considered SSL CA’s as a potential victim
set, since they had use Domain Validation to
– Microsoft helped us contact all of those
• Finally, we considered autoupdaters, but frankly
we didn’t think anything popular would be that
– Got LinkedIn fixed
– Amato demonstrated this angle pretty
Why a web based tester – that doesn’t
let you choose which server to test?
• 1) People have no idea what name servers they use
– It’s set once and forgotten about
– The servers can never die, so they keep accumulating
new random users
• Use is almost viral
– By using whatever DNS server is enabling the browser
(or eventually, the mail server), users could measure
their actual exposure to an attack
• 2) Users are customers, and customer awareness was a key
driver of getting fixes adopted
– Alternately, customers could protect themselves by
configuring their own DNS
• Or OpenDNS
Loop de loop
• I wanted to operate within a single resolution thread
– DNS allows aliases – “CNAMEs” – that force another
– I statelessly encoded IP, Port, and TXID into each hop
– At 5 hops, decode and save a file for a web server to
• $foo = rand();
• $ curl http://1234.doxdns1.com/fprint/1234
1234.doxdns5.com,Sat Dec 27 20:17:50
A Small Testing Bug
• Originally, all five requests went to the same name
– Unfortunately, some fixes would randomize a
port per destination IP
– Ports would stay the same as long as the same
IP was in use, which obviously wasn’t an attack
– Repaired that by putting each CNAME in a
different domain, off a different name server
(doxdns1.com, doxdns2.com, etc.)
– Paul Vixie / ISC hosted this
Bringing Mail Into The Picture
• Needed to allow people to send mails, to
find out which DNS server their mail server
– Generated a random mail address, then
– Do not try to wrap your mind around the
HTTP to SMTP to DNS to HTTP to
HTML backchannel this represents
Left Undone: Embedding
• Considered allowing people to embed a
tester into their own blogs and websites
– Actually was asked for this
• Problem: No major site would ever support
www.doxpara.com/dnstest.js>. Not going
• Solution: Billy Hoffman
• Web browsers are very restrictive about what information they’ll
allow to flow cross domain
• However, there’s one small channel that doesn’t allow scripting
across sites, but does allow a bit of information to flow, and works
on all browsers
– Image size!
• “I loaded this image from a foreign domain. What are its
– I found this independently, but Billy Hoffman mastered it for full
data transmission (little slow)
• If dimensions were 1x1, then DNS was safe, 1x2, then DNS unsafe,
• Probably the first good use for the Image Size side channel – cross
domain information retrieval from untrusted sites on all browsers
So Now What
• We’re at about 75%
– Probably not getting much better anytime soon
– No numbers (yet) on what % of users are
covered by that 75%
• Could be more, could be less
• Most major ISP’s are patched though
• Do we keep going?
– Yes. All the bugs are still there, they’re just
masked – and not well – by this one DNS fix
DNS Is Not Alone
• There have been (at least) four bugs this year that have
exposed content that is sadly trusting its gateway
• There will be more.
• I said earlier, I believed DNSSEC can help
– The question is not why DNS was insecure. The
question was why everything else was, and what to do
A Few Thoughts on DNSSEC
• The present numbers say nothing.
– DNSSEC, like all authoritative-server modifying
solutions, needs the root signed for the solution
to be meaningful
• Otherwise, the attacker just attacks the
• XQID thought they got around this. Bug me
if you want to see the break in XQID.
– The root has remained unsigned for far too
long. That’s apparently going to change.
• Apparently, the big thing in the DNS world right
now is “automation” – how do we make key
signing just happen?
– DNS scales. DNSSEC in its “normal operating
modes” requires a fairly extensive amount of
– Yeah, no. DNSSEC needs to be as close to a
deployment story as the SPR patch as
Integration With Registries and
• DNS is the only successful federated technology.
• DNSSEC solves the problem of getting data back
– The registries and registrars are the
human/business factors that get data in
– Easing the business load on them is as
important as making DNSSEC manageable for
the end administrator
• Regarding DNSCurve, I think we have a lot
to learn from it
– DNSCurve is DJB’s concept for how to
• It’s based on link-based crypto instead
of anything that can be cached
• What’s Good
– It posits online key signing
• DNS material is far too dynamic, and admins
are far too harried, for the old model of the
offline keystore to make sense
– Registrars don’t have much to do – chaining is
handled by the names of name servers
• What’s not so good
– There’s no code.
• Um, that matters.
– It requires new crypto.
• ECC is standard, but the proposed curve is
• “Optimized for speed” is not actually what
you want to hear about a cryptosystem.
– It’s not actually that fast.
DNSCurve and Performance
• DNSSEC was designed to require no per-query crypto
operations on the servers, which may be heavily loaded
– All operations may be done once, and cached
• DNSCurve does a crypto operation per query
– With DJB’s sample code, a laptop that can do 15,000
DNS queries a second can do maybe 10,000 ECC
operations per second. With 1 operation inbound and 1
operation outbound, that’s 100% CPU on 1/3rd
before you’ve parsed a single DNS packet
• Could possible be optimized, but why?
The Big Problem
• There’s no way to achieve end-to-end trust with DNSCurve.
– With DNSSEC, eventually we can envision clients that do
their own validation, using the name server infrastructure
just to cache
– DNSCurve offers a choice: Either abandon end-to-end
trust (stub resolver doesn’t talk to the real heirarchy), or
abandon caching (stub resolver does talk to the
• The DNS cannot absorb a 100x increase in load,
even without added CPU hit from the crypto.
• Again, DNSCurve has some really cool
ideas for how to make DNS more secure.
• We have more to learn from DJB!
• 1) Federation is hard, and everyone has been
using DNS to do it. That DNS isn’t secure, and
that people needed DNS to be secure, wasn’t
actually enough to stop people from using the only
workable federation scheme out there.
• 2) Authentication in general is a mess.
• 3) If DNS is to be at the heart of authentication,
then perhaps it should be more secure.
• 4) DNSCurve offers some cool possibilities for
making DNSSEC better.
One More Thing…
• Remember when I polluted doxpara.com,
so that I could collect the password from
I also polluted backend.doxpara.com. We
REALLY need to fix DNS.