The document discusses logging best practices for incident response in a post-IPv4 internet where Carrier Grade NATs (CGNs) are commonly used. It notes that with CGNs, a single public IPv4 address can represent thousands of users, meaning the source IP address alone is no longer enough to identify individual attackers. It recommends logging additional details like source ports and high-resolution timestamps to aid incident response. It provides examples of configuring Apache and Exim servers to log source ports. It also stresses the importance of network time synchronization across logging systems.

1. Logging for Incident Response in the
Post-IPv4 World
Carlos Martinez Cagnazzo
LACNIC
carlos @ lacnic.net
@carlosm3011

2. Agenda
• The Post-IPv4 Internet
– No IPv4, CGNs, some IPv6
• Logging for incident response
• Logging and incident response in the post-IPv4 Internet

3. The Post-IPv4 Internet
• The Internet is at a crossroads. IPv4 exhaustion means
that there will not be enough IPv4 addresses for every
one, much less for every device
• To an extent, this is already happening, but from now it
will the norm
120
100
80
60
40
20
0
1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011
…

4. The Current, Almost End-to-End, Internet
• Once upon a time there was something called the ‘End
to End Principle’
– … describing how packets should travel from origin to
destination untouched by the evil middle boxes
• The current Internet is _almost_, but not quite, end-toend
– Proxies, home routers, firewalls, traffic shapers, all of them do
something to packets
– But packets travel mostly unharmed

5. The Current End-to-End Internet
• Well, almost end to end**
D_Addr | O_Addr | Payload
D_Addr | O_Addr | Payload
• Packets remain (mostly) unchanged along their network
path
• A given source IP can be a marker of an individual, a
household or an employee of a certain company

6. What happens when there is no IPv4 for every device ?
• The post-IPv4 Internet:
Single public
IP address
Web server sees
thousands of
users coming
from the
*same* IP
• IPv4 will be provided, in many places, by employing
CGNs, or Carrier-Grade NAT boxes

7. The CGN-ized Internet
• The CGN Internet hides many users behind a small set of
IP addresses
• Our previous assumptions about what a source IP
address means are no longer valid
– Can represent thousands of users, of different households and
different companies
• Many abuse mitigation measures need to be reexamined
– Be careful of blankly filtering out a single /24, that could now
mean 10.000 users

8. Current practice for Incident Response
• Think for a minute about your usual IR workflow
– Phishing, Spam, DDoSing, you name it
• When your incident involves network traffic, you try to
find the following information:
–
–
–
–
Source IP addresses
Destination IP addresses and destination ports
Maybe a packet dump, if available
All of this decorated with nice timing information, preferable
with a common time zone
• You then look the sources in WHOIS or in your friendly
CSIRT contact list and send the appropriate notifies

9. The Post-IPv4 Incident Response Workflow
• Well, source IPv4 address may not be enough of an
identifier anymore
– The source network will not be able to identify the actual
offender(s) just based on the source IPv4 address
• ISPs will need source port data to actually track any
abusers
• Law enforcement also needs to realize what this means
– Judges now need to look at an additional number before jailing
a person

10. Jeez, what do we do now ?
• First of all, accept that now your life as an incident
response or site administrator will be harder
– Hopefully for a short time, until the world gets its IPv6 act
together
• Additional requirements for post-IPv4 logging
– Logging of source ports
– Using the highest possible timing resolution
– Time sync on distributed logging platforms becomes critical

11. Example configuration, Source Port Logging in Apache
• [Ref: http://draft.scyphus.co.jp/articles/20110815.html]
– Default logging in Apache only provides basic client data
– Apache uses a printf()-like format for including additional log
fields in custom log files
#
# The following directives define some format nicknames for use
# with a CustomLog directive (see below).
#
LogFormat "[%h]:%{remote}p %l %u %t "%r" %>s %b
"%{Referer}i" "%{User-Agent}i"" combined
LogFormat "[%h]:%{remote}p %l %u %t "%r" %>s %b" common

12. Example configuration, Source Port Logging in Apache
• [Ref: http://draft.scyphus.co.jp/articles/20110815.html]
– Default logging in Apache only provides basic client data
– Apache uses a printf()-like format for including additional log
fields in custom log files
<VirtualHost [2001:13c7:7001:4000::10]:80>
ServerAdmin carlos@lacnic.net
DocumentRoot /var/www/html/
ServerName w6.labs.lacnic.net
LogFormat "[%h]:%{remote}p %l %u %t "%r" %>s %b
"%{Referer}i" "%{User-Agent}i"" combined
CustomLog logs/w6.labs.lacnic.net-access_log
"[%h]:%{remote}p %l %u %t "%r" %>s %b "%{Referer}i"
"%{User-Agent}i""
# LogFormat "[%h]:%{remote}p %l %u %t "%r" %>s %b" common
ErrorLog logs/w6.labs.lacnic.net-error_log

13. Example configuration, Source Port Logging in Apache
• Must enable “mod_log_config” if not already enabled
[2001:13c7:7003:89:fcda:8bea:3e8a:cedd]:57366 - [31/Oct/2013:15:01:33 -0200] "GET
/site/modules/openid/openid.js?A HTTP/1.1" 304 "http://w6.labs.lacnic.net/site/" "Mozilla/5.0 (Macintosh; Intel
Mac OS X 10_9_0) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/30.0.1599.101 Safari/537.36"
[2001:13c7:7003:89:fcda:8bea:3e8a:cedd]:57365 - [31/Oct/2013:15:01:33 -0200] "GET
/site/themes/newlabs/print.css?A HTTP/1.1" 304 "http://w6.labs.lacnic.net/site/" "Mozilla/5.0 (Macintosh; Intel
Mac OS X 10_9_0) AppleWebKit/537.36 (KHTML, like Gecko)
Chrome/30.0.1599.101 Safari/537.36”

14. Example configuration, Exim4 logging
• Sample configuration:
– [ http://www.exim.org/exim-htmlcurrent/doc/html/spec_html/ch-log_files.html ]
# uncomment this for debugging
# MAIN_LOG_SELECTOR == MAIN_LOG_SELECTOR +all subject -arguments
.ifdef MAIN_LOG_SELECTOR
log_selector = MAIN_LOG_SELECTOR +incoming_port
.endif
2013-10-28 17:22:17 1VasOD-0005hG-KT <= carlos@lacnic.net
H=localhost (coco) [127.0.0.1]:47264 P=esmtp S=474
2013-10-28 17:22:17 1VasOD-0005hG-KT => marcelo
<marcelo@localhost> R=local_user T=maildir_home
2013-10-28 17:22:17 1VasOD-0005hG-KT Completed

15. Distributed logging
• Did I say ‘time sync’ before ? 
• Use NTP Luke, You Must.
– It was invented for a reason
• Look into fast data stores and mining tools
– Splunk
– ElasticSearch
– NoSQL databases (Redis, MongoDB)

16. Key Takeaways
• Yes, our sys/netadmin life will be harder, at least until
IPv6 is widely deployed
– Let’s embrace it with a smile
• Do not assume that a source attack IPv4 address
uniquely identifies an attacker anymore
– Or a victim, in some cases, like phishing sites
• Start logging source ports now. If you are a CSIRT, do not
forget to reach out to your constituency and let them
know this
• Send source ports when reporting incidents. Ask for
source ports when receiving incident reports

17. Key Takeaways (ii)
• Log with the highest timing resolution your equipment
allows
• And repeat with me…
– I will time sync my systems
– I will time sync my systems
– I will time sync my systems

18. Thank you very much! Questions?
@carlosm3011