Jaime Piña, @variadico, Software Engineer at Apcera Microservice issues are networking issues. Fixing code in your app is easy, but the hard part of using microservices is the networking. How do you actually know if you're sending what you think you are? Why does this request fail in my app, but not when I use curl? Is this service very slow or is it up at all? This talk will help demystify some common problems you might experience while building out your collection of microservices. Once you can find the issue, it becomes way easier to fix.

1. Debugging Network
Issues
4 things learned the hard way and by asking all my friends a bunch of
questions.

2. Who is this guy?
● Jaime Piña
● Software engineer at Apcera
● Apcera platform deploys apps to hybrid cloud with policy
● Work on Apcera Setup, gateways, and other things
(Hai-meh)

3. Architecture (simplified)

4. Microservices
sit on the
network.
One does not simply
use the network.

5. Is the server plugged
in?
(Is your app running?)

6. Connection refused!
$ ssh jaime@localhost
ssh: connect to host localhost port 22: Connection refused
$ curl http://localhost
curl: (7) Failed to connect to localhost port 80: Connection refused

7. Is there a firewall?

8. Connection refused! (Part 2)
$ curl http://1.2.3.4
curl: (7) Failed to connect to 1.2.3.4 port 80: Connection refused

9. Server side check with ufw
# ufw status
Status: active
To Action From
-- ------ ----
22 ALLOW Anywhere

10. Connection refused! (Part 2)
$ curl http://1.2.3.4
curl: (7) Failed to connect to 1.2.3.4 port 80: Connection refused

11. Server side check with iptables
A little more complicated.

12. iptables vocabulary
● INPUT chain
○ Incoming
● OUTPUT chain
○ Outgoing
● ACCEPT action
○ Allow
● DROP action
○ Block

13. iptables --list
Chain INPUT (policy DROP)
target prot opt source destination
ufw-user-input all -- anywhere anywhere
Chain ufw-user-input (1 references)
target prot opt source destination
ACCEPT tcp -- anywhere anywhere tcp dpt:ssh
ACCEPT udp -- anywhere anywhere udp dpt:ssh

14. Connection refused! (Part 2)
$ curl http://1.2.3.4
curl: (7) Failed to connect to 1.2.3.4 port 80: Connection refused

15. Don’t have server access?

16. Client side check with nmap
$ nmap scanme.nmap.org
Not shown: 971 closed ports
PORT STATE SERVICE
22/tcp open ssh
5269/tcp filtered xmpp-server
6007/tcp filtered X11:7

17. nmap vocabulary
● open state
○ Port accessible, app listening
● closed state
○ Port accessible, no app listening
● filtered state
○ IDK? ¯_(ツ)_/¯

18. Client side check with nmap
$ nmap scanme.nmap.org
Not shown: 971 closed ports
PORT STATE SERVICE
22/tcp open ssh
5269/tcp filtered xmpp-server
6007/tcp filtered X11:7

19. Moar logs!
(And verbose output!)

20. Triple verbose SSH
$ ssh -vvv jaime@foo.com

21. We can reach the server
debug2: resolving "foo.com" port 22
debug1: Connecting to ejemplo.com [1.2.3.4] port 22.
debug1: Connection established.

22. Trying to read my public key
debug1: key_load_public: No such file or directory
debug1: identity file /home/jaime/.ssh/id_rsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/jaime/.ssh/id_ed25519 type -1

23. Trying to use key auth
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Trying private key: /home/jaime/.ssh/id_rsa
debug3: no such identity: /home/jaime/.ssh/id_rsa: No such file
or directory
debug1: Trying private key: /home/jaime/.ssh/id_ed25519
debug3: no such identity: /home/jaime/.ssh/id_ed25519: No such
file or directory

24. Trying to use password auth
debug1: Next authentication method: password
jaime@foo.com's password:

25. systemd
journalctl --unit ssh.service

26. Careful with DNS

27. DNS
The thing that turns English
words into numbers.
Input: www.google.com
Output: 172.217.6.36

28. The thing is
DNS is not required for working internet.

29. One day...
# rkt run --insecure-options=image --interactive docker://ubuntu:14.04
root@rkt:/# apt-get update
Err http://archive.ubuntu.com trusty-updates InRelease
Err http://archive.ubuntu.com trusty-security InRelease
Err http://archive.ubuntu.com trusty-updates Release.gpg
Could not resolve 'archive.ubuntu.com'
Err http://archive.ubuntu.com trusty-security Release.gpg
Could not resolve 'archive.ubuntu.com'
Reading package lists... Done
W: Failed to fetch
http://archive.ubuntu.com/ubuntu/dists/trusty-updates/InRelease
W: Failed to fetch
http://archive.ubuntu.com/ubuntu/dists/trusty-security/InRelease

30. What’s happening??

31. Is the internet cable
plugged in?

32. What’s happening??

33. Closer look
Err http://archive.ubuntu.com trusty-updates Release.gpg
Could not resolve 'archive.ubuntu.com'
Err http://archive.ubuntu.com trusty-security Release.gpg
Could not resolve 'archive.ubuntu.com'
Reading package lists... Done
W: Failed to fetch
http://archive.ubuntu.com/ubuntu/dists/trusty-updates/InRelease
W: Failed to fetch
http://archive.ubuntu.com/ubuntu/dists/trusty-security/InRelease

34. Test for success
root@rkt:/# ping -c 1 172.217.6.36
PING 172.217.6.36 (172.217.6.36) 56(84) bytes of data.
64 bytes from 172.217.6.36: icmp_seq=1 ttl=49 time=26.2 ms
--- 172.217.6.36 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 26.297/26.297/26.297/0.000 ms

35. Test for failure
root@rkt:/# ping -c 1 www.google.com
ping: unknown host www.google.com

36. OMG it’s the DNS!

38. Fix
rkt run --insecure-options=image --dns 8.8.8.8
--interactive docker://ubuntu:14.04

39. Worked.

40. How to get DNS info?
drill (or dig)

41. drill usage
drill name [@nameserver] [record type]

42. DNS vocabulary
● nameserver
○ Server who has DNS info about a domain
● A record
○ Contains IP address for a domain
● NS record
○ Contains nameservers for a domain

43. drill example
$ drill www.google.com @8.8.8.8 A
;; QUESTION SECTION:
;; www.google.com. IN A
;; ANSWER SECTION:
www.google.com. 297 IN A 172.217.6.36

44. Packet inspection
(I do not think you’re sending what you
think you’re sending.)

46. tcpdump syntax
tcpdump [options] 'BPF'

47. tcpdump basic usage
tcpdump -i lo 'BPF'
More: tcpdump --list-interfaces

48. Berkeley Packet Filter
● host foo.com
○ src/dst host is foo.com
● tcp port 22
○ TCP traffic coming/going to port 22
● dst port 53
○ Traffic going to port 53

49. tcpdump example
# tcpdump -i wlp58s0 'tcp port 80'
192.168.0.109.37370 > 107.170.18.175.http: Flags [S], length 0
107.170.18.175.http > 192.168.0.109.37370: Flags [S.], length 0
192.168.0.109.37370 > 107.170.18.175.http: Flags [.], length 0
Flags:
S = SYN . = ACK
Me -SYN-> server
Me <-SYN ACK- server
Me -ACK-> server

50. World Famous
Three-way Handshake

52. tcpdump flag Flag name Description
S SYN Signal start of connection
. ACK Acknowledge packet
P PSH Sending data
F FIN Signal end of connection
R RST Connection killed
Common packet flags

53. ngrep syntax
ngrep [options] 'pattern' 'BPF'

54. ngrep example
# ngrep -q -Wbyline "HTTP" "tcp port 80"
T 192.168.0.109:59990 -> 107.170.18.175:80 [AP]
POST /portfolio/wp-login.php HTTP/1.1.
Accept-Encoding: gzip, deflate.
.
log=HELLO&pwd=WORLD&wp-submit=Log+In&redirect_to=http%3A%2F%2Ffoo.c
om%2Fportfolio%2Fwp-admin%2F&testcookie=1

55. Credentials
log=HELLO&pwd=WORLD

56. How to debug (some) network issues
● Is your app running?
● Is there a firewall?
● Does the DNS work?
● Are you sending and receiving what you think you
are?

57. Thanks
Jaime Piña
Software engineer at Apcera
@variadico