The talk will start explaining how Tor project can help us to the research and development of tools for online anonymity and privacy of its users while surfing the Internet, by establishing virtual circuits between the different nodes that make up the Tor network. In addition, we will review how Tor works from anonymity point of view, preventing websites from tracking you. Python help us to automate the process to search an discover hidden services thanks to packages like requests,requesocks and sockspy,At this point we will review the crawling process and show tools in python ecosystem available for this task(https://github.com/jmortega/python_dark_web) These could be the talking points: -Introduction to Tor project and hidden services -Discovering hidden services. -Modules and packages we can use in python for connecting with Tor network -Tools that allow search hidden services and atomate the crawling process in Tor network

1. www.sti-innsbruck.at
@jmortegac May,2019
DARKWEB + PYTHON: DISCOVER,
ANALYZE AND EXTRACT INFORMATION
FROM HIDDEN SERVICES

2. About me
2
http://jmortega.github.io/

3. About me
3

4. Agenda
• Introduction to Tor project and hidden
services
• Discovering hidden services
• Modules and packages we can use in
python for connecting with Tor network
• Tools that allow search hidden services
and atomate the crawling process in Tor
network
4

5. Surface vs Deep vs Dark Web
5

6. What is Tor?
6
• Tor is a free tool that allows people to use the
internet anonymously.
• Tor anonymizes the origin of your traffic

7. What is Tor?
7

8. What is Tor?
8

9. Onion Routing
9
Tor is based on Onion Routing, a technique for
anonymous communication over a computer network.

10. 10
Onion Routing

11. 11
User's software or client
incrementally builds a
circuit of encrypted
connections through
relays on the network.
Establish TOR circuit

12. 12
When we connect to
the TOR network, we
do it through a circuit
formed by 3
repeaters, where the
encrypted packet sent
from the client is
passing. Each time
the packet goes
through a repeater, an
encryption layer is
added.
Establish TOR circuit

13. 13
User's software or client
incrementally builds a
circuit of encrypted
connections through
relays on the network.
Hidden services

14. Hidden services
14
https://metrics.torproject.org/hidserv-dir-onions-seen.html

15. Tor NODE List
15

16. Tor NODE List
16
https://www.dan.me.uk/tornodes
http://torstatus.blutmagie.de

17. Tor NODE List
17
https://onionite.now.sh

18. Exonera TOR
18
https://metrics.torproject.org/exonerator.html

19. Relay search
19
https://metrics.torproject.org/rs.html#simple

20. Relay search
20
https://metrics.torproject.org/rs.html#simple

21. Relay search
21
https://metrics.torproject.org/rs.html#simple

22. Discover hidden services
22
HiddenWiki:http://wikitjerrta4qgz4.onion/
Dark Links: http://wiki5kauuihowqi5.onion
Tor Links: http://torlinkbgs6aabns.onion
Dark Web Links:
http://jdpskjmgy6kk4urv.onion/links.html
HDWiki: http://hdwikicorldcisiy.onion
OnionDir: http://dirnxxdraygbifgc.onion
DeepLink: http://deeplinkdeatbml7.onion
Ahmia: http://msydqstlz2kzerdg.onion

23. Tor onnion services
23

24. Tor onnion services
24
https://en.wikipedia.org/wiki/List_of_Tor_onion_
services
https://en.wikipedia.org/wiki/The_Hidden_Wiki

25. TOR2web
25
https://www.onion.to

26. TOR browser
26
https://www.torproject.org/download/

27. 27
Onion Routing

28. Installing TOR
28
sudo apt-get update
sudo apt-get install tor
sudo /etc/init.d/tor restart

29. TORrc
29

30. Running TOR
30
$ tor --SocksPort 9050 --ControlPort 9051

31. Running TOR
31

32. Tor service
32
service tor start/restart
service tor status

33. Connecting with TOR
33
Stem
https://stem.torproject.org/
TorRequest
https://github.com/erdiaker/torrequest
Requests + socks5

34. Stem
34
pip install stem

35. TOR descriptors
35
Server descriptor: Complete information about a repeater
ExtraInfo descriptor: Extra information about the repeater
Micro descriptor: Contains only the information necessary for
TOR clients to communicate with the repeater
Consensus (Network status): File issued by the authoritative
entities of the network and made up of multiple entries of
information on repeaters (router status entry)
Router status entry: Information about a repeater in the
network, each of these elements is included in the consensus
file generated by the authoritative entities.

36. TOR spec
36

37. Stem
37
from stem import Signal
from stem.control import Controller
with Controller.from_port(port = 9051) as
controller:
controller.authenticate(password='your
password set for tor controller port in torrc')
print("Success!")
controller.signal(Signal.NEWNYM)
print("New Tor connection processed")

38. Periodic Tor IP Rotation
38
import time
from stem import Signal
from stem.control import Controller
def main():
while True:
time.sleep(20)
print ("Rotating IP")
with Controller.from_port(port = 9051) as controller:
controller.authenticate()
controller.signal(Signal.NEWNYM) #gets new identity
if __name__ == '__main__':
main()

39. Stem.Circuit status
39
from stem.control import Controller
controller = Controller.from_port(port=9051)
controller.authenticate()
print(controller.get_info('circuit-status'))

40. Stem.Network status
40
from stem.control import Controller
controller = Controller.from_port(port=9051)
controller.authenticate(password)
entries = controller.get_network_statuses()
for routerEntry in entries:
print(routerEntry)

41. Stem.circuits
41

42. Stem.circuits
42

43. Server descriptors
43

44. Introduction points
44

45. Tor nyx
45
https://nyx.torproject.org/

46. Tor nyx
46

47. Tor nyx
47

48. Tor nyx
48

49. TorRequest
49
from torrequest import TorRequest
with TorRequest() as tr:
response = tr.get('http://ipecho.net/plain')
print(response.text) # not your IP address
tr.reset_identity()
response = tr.get('http://ipecho.net/plain')
print(response.text) # another IP address

50. Request
50
import requests
def get_tor_session():
session = requests.session()
# Tor uses the 9050 port as the default socks port
session.proxies = {'http': 'socks5h://127.0.0.1:9050',
'https': 'socks5h://127.0.0.1:9050'}
return session
# Following prints your normal public IP
print(requests.get("http://httpbin.org/ip").text)
# Make a request through the Tor connection
# Should print an IP different than your public IP
session = get_tor_session()
print(session.get("http://httpbin.org/ip").text)
r = session.get('https://www.facebookcorewwwi.onion/')
print(r.headers)

51. Analyze hidden services
51
1) Queries to the data sources.
2) Filter adresses that are active.
3) Testing against each active address and
analysis of the response.
4) Store URLs from websites.
5) Perform a crawling process against each
service
6) Apply patterns and regular expressions to
detect specific content(for example mail
addresses)

52. Ahmia search engine
52
https://ahmia.fi/

53. Torch search engine
53
http://xmh57jrzrnw6insl.onion

54. UnderDir Search engine
54

55. Hidden services
55

56. Search Hidden services
56

57. Other tools
57
POOPAK - TOR Hidden Service Crawler
https://github.com/teal33t/poopak
Tor spider
https://github.com/absingh31/Tor_Spider
Tor router
https://gitlab.com/edu4rdshl/tor-router

58. Onnion scan
58
https://github.com/s-rah/onionscan

59. Dark Web map
59
https://www.hyperiongray.com/dark-web-map/

60. GitHub repositories
https://github.com/serfer2/python-deepweb
https://github.com/jmortega/python_dark_web
60