Meeting #4.

1. Defcon-Moscow, 23/11/13
research
Hashcracking server on
generic hardware
Ivan Novikov (@d0znpp)

2. Cracking hashes? For what?
• Web application security audits: whitebox
and blackbox
• Passwords -> hashes
• Crackings hashes to gain access

3. 2009-2011 - we used CPU on our laptops
2011 - first hashcracker, named “Parovoz”
based on Bulldozer + HD6990
2013 – parovoz-ng – HD6990 + 2xHD7990 –
up to 7xPCIEx16 cards + water cooling
History

4. GPU for hashcracking
Nvidia or AMD ?

5. Win7 + gtx580 Win7+HD7970 Ubuntu+HD6990 Ubuntu+gtx560ti
www.hashcat.net/oclhashcat-plus

6. Just INTEGER for AMD ;)
Float calculation – Nvidia
Why HD6990 faster than HD7970?
Why AMD faster?

7. Depends from software…
JohnTheRipper – opensource, both
oclHashcat-plus/lite – free, bin, UNIX
Extreme GPU bruteforcer – bin, Win, Nvidia
only (CUDA)
Win or Linux ?

8. Debian linux
oclHashcat-plus for every day using
oclHashcat-lite for single hash cracking
JohnTheRipper for rules attacks, rare hashes,
when hashcat not working ;)
Where we left of

9. Configure X11 to use a few cards at the same
time (thx @defconua)
Install Catalyst driver (.run file with black
magic)
Find combination of oclHashcat and Catalyst
versions which can works together
Monitoring/reports/etc on unstable hardware
Common software problems

10. • Cheaper than server platform (starts from
$5000 / platform)
• Easier to reproduce
• Occupies less space
• Funniest ;)
Generic hardware – for what?

11. First!
You need a big tower!

13. HD6990 - $700 – not in retail now
HD7990 - $1000 – easy to buy
Motherboard –
from $100 (1xPCIEx16)
to $500 (7xPCIEx16)
Smth else? :)
What about costs

14. Cooling
COOOOOLING !!!
Power supply
POOOOOWER SUUUPLYY !!!
Common hardware problems

15. One motherboard – one ATX connector
Each power supply need ATX connector to
work
We want to use more that one power supply
but only one motherboard!
How to build multi power supply
system on general hardware?

16. How to build multi power
supply system on
general hardware?
 easy hack ;)

17. ~ 500W per card (HD[6|7]990)
Seller: 1300W power supply
2 x HD7990 on 1300W PS and… F*ck off!!!
Why?
How to choose power supply?

18. 1200W on power supply but…
How many channels? (1/2)
How many power per channel? (less than
500W ;-( )
How to choose power supply?

19. Default air cooling on retail cards
HD6990 – good! Works well on 99% load,
stock clocks, about 87 C temp
HD7990 – bad ;( Works on 99% load, stock
clocks, about 20 seconds (than 99 C temp)
Cooooooling

20. Water pump - $100-200 1-2 for system
Water block - $150-200 for each card
Radiator - $100-200
Other supply - $150 (hoses, water, others)
Water cooling – only one way ;(

21. Water pump - $100-200 1-2 for system
Water block - $150-200 for each card
Radiator - $100-200
Other supply - $150 (hoses, water, others)
Water cooling – only one way ;(

22. Aircooling
• 20 seconds to 97C on HD7990
• 87C on HD6990 stable
Water cooling:
• 60 C stable on HD6990 and HD7990 both
Water cooling – results

23. <- Waterblock
Radiator ->

25. Current:
• $5000 for all – 42G MD5/s (42*10^10
hashes per second)
• $119/1G MD5
Target:
• $9000 for all – 112G MD5/s
• $80/1G MD5
Our results

26. Double Hawaii core card +40%
Overclock of Hawaii +30%
Total about:
$9000 for 216.5G MD5/s
$41/1G MD5
Optimistic target

27. The end
Contacts:
@wallarm, @d0znpp
research