More Related Content Similar to BIND 9 logging best practices (20) More from Men and Mice (20) BIND 9 logging best practices1. © Men & Mice http://menandmice.com
Webinar
BIND 9 logging best practices
1
2.2.2017
2. © Men & Mice http://menandmice.com
Agenda - BIND 9 logging
BIND 9 logging configuration
The BIND 9 logging categories
Query- and Debug-Logging
A BIND 9 logging configuration template
Using the logeater tools
Filtering Logs with artificial ignorance
2
3. © Men & Mice http://menandmice.com
BIND 9 logging configuration
3
4. © Men & Mice http://menandmice.com
BIND 9 logging
Compared with other Unix-Daemons, BIND 9 has a sophisticated log
infrastructure built in.
It contains:
•several destinations for log information (channel)
•logging to syslog, stdout, files
•several categories
•syslog severities
•date/timestamp
•automatic size-based log-file rotation
4
5. © Men & Mice http://menandmice.com
BIND Logging
Channel
(where to log)
syslog
file /var/named/query.log
file /var/named/transfer.log
null
stdout
Category
(what to log)
security
xfer-in / xfer-out
queries
dnssec
default
5
6. © Men & Mice http://menandmice.com
The Default Category
The default category defines the logging for categories where
no specific configuration has been defined
The queries category is an exception, and not logged
through default.
6
Category
(what to log)
default
7. © Men & Mice http://menandmice.com
Example Logging Configuration
logging {
channel syslog { syslog daemon; severity info; };
channel stdout { stderr; severity info; };
channel transfer_log {
file "/var/named/transfer.log" versions 10 size 10M;
severity info;
print-time yes; print-severity yes; print-category yes;
};
channel query_log {
file "/var/named/query.log" versions 10 size 50M;
severity debug;
print-time yes;
};
category security { syslog; };
category queries { query_log; };
category xfer-in { transfer_log; };
category xfer-out { transfer_log; };
category dnssec { syslog; };
category default { syslog; stdout; };
};
7
Channel names have no
special meaning (this includes
syslog and stdout in the
example).
By default, file size is not
limited, and file rotation
(versions) are not made.
The default severity is info.
By default, the time, severity
and category are not logged.
8. © Men & Mice http://menandmice.com
Predefined Channels
logging {
channel syslog { syslog daemon; severity info; };
channel stdout { stderr; severity info; };
channel transfer_log {
file "/var/named/transfer.log" versions 10 size 10M;
severity info;
print-time yes; print-severity yes; print-category yes;
};
channel query_log {
file "/var/named/query.log" versions 10 size 50M;
severity debug;
print-time yes;
};
category security { default_syslog; };
category queries { query_log; };
category xfer-in { transfer_log; };
category xfer-out { transfer_log; };
category dnssec { default_syslog; };
category default { default_syslog; default_stderr; };
};
8
There are four predefined
channels:
default_syslog,
default_stderr,
default_debug,
null.
default_syslog and
default_stderr are identical to
the channels in the example,
making those unnecessary.
default_debug writes to the file
named.run in BIND's working
directory.
9. © Men & Mice http://menandmice.com
Logging Defaults
If named.conf has no logging statement, BIND essentially
defaults to use syslog.
The default configuration is:
logging {
category default { default_syslog; default_debug; };
category unmatched { null; };
};
The BIND 9 process can be started with a default log-file. The
default category will then be logged into this file instead to syslog:
# named -L /var/log/bind9.log
9
10. © Men & Mice http://menandmice.com
BIND 9 logging categories
10
11. © Men & Mice http://menandmice.com
logging categories (1)
Below is a list of available logging categories in BIND 9
11
https://ftp.isc.org/isc/bind9/cur/9.11/doc/arm/Bv9ARM.pdf
Category Description
default
The default category defines the logging options for those categories
where no specific configuration has been defined.
client Processing of client requests.
cname
Logs nameservers that are skipped due to them being
a CNAME rather than A / AAAA records.
config Configuration file parsing and processing.
12. © Men & Mice http://menandmice.com
logging categories (2)
12
https://ftp.isc.org/isc/bind9/cur/9.11/doc/arm/Bv9ARM.pdf
Category Description
database
Messages relating to the databases used internally by the name server
to store zone and cache data.
delegation-only
Logs queries that have been forced to NXDOMAIN as the result of a
delegation-only zone or statement.
declaration
dispatch
Dispatching of incoming packets to the server modules where they
are to be processed.
dnssec DNSSEC and TSIG protocol processing.
dnstap The "dnstap" DNS traffic capture system.
edns-disabled
Log queries that have been forced to use plain DNS due to timeouts.
See BIND 9 ARM for additional information.
general
The catch-all. Many things still aren’t classified into categories, and
they all end up here
lame-servers Lame servers. These are misconfigurations in remote servers.
13. © Men & Mice http://menandmice.com
logging categories (3)
13
https://ftp.isc.org/isc/bind9/cur/9.11/doc/arm/Bv9ARM.pdf
Category Description
network Network operations.
notify
The NOTIFY protocol. Used to inform servers of slave-copies of a
zone about changes.
queries Query-Logging. See below.
query-errors Information about queries that resulted in some failure.
rate-limit DNS response rate limiting. Logs start- and end-times of limiting.
resolver
DNS resolution, such as the recursive lookups performed on behalf of
clients by a caching name
server.
rpz
Information about errors in response policy zone files, rewritten
responses.
security Approval and denial of requests. ACL violations.
14. © Men & Mice http://menandmice.com
logging categories (4)
14
https://ftp.isc.org/isc/bind9/cur/9.11/doc/arm/Bv9ARM.pdf
Category Description
spill
Logs queries that have been terminated, either by dropping or
responding with SERVFAIL, or as a result of a fetchlimit quota.
unmatched
Messages that named was unable to determine the class of or for
which there was no matching view.
update Dynamic updates.
update-security Approval and denial of update requests.
xfer-in Zone transfers the server is receiving.
xfer-out Zone transfers the server is sending.
15. © Men & Mice http://menandmice.com
Query Logging
•Query logging generates one line for each query
received, and can be toggled on/off with: rndc querylog
•Logging queries slows BIND down significantly.
•Use it for troubleshooting.
•A server receiving 100 qps (queries/sec) handles logging.
•Tests show a server with 400 kqps drops to 50%
performance with query logging enabled.
15
16. © Men & Mice http://menandmice.com
Query Logging
31-Jan-2016 21:39:27.784 client ::1#42482 (alpha.zone02.dnslab.org):query:
alpha.zone02.dnslab.org IN A +ET (::1)
31-Jan-2016 21:48:30.172 client 192.168.53.251#7691 (zone02.dnslab.org):
query: zone02.dnslab.org IN NS -EDC (192.168.53.102)
client's IP address
and port
domain name
queried
record type
queried
Flags:
(+) = recursive query (RD flag)
(-) = iterative query (no RD flag)
(E) = EDNS used in query
(T) = query over TCP
(D) = DO (DNSSEC Ok)
(C)= Checking Disabled
(S) = Signed Query
Server's address
query arrived on
16
17. © Men & Mice http://menandmice.com
Debug-Logging
•trace <level>: Change BIND's debugging level to the
file "named.run" (Reading output requires some
knowledge of BIND internals.)
rndc trace 10
rndc trace
rndc notrace
rndc trace 0
•The debugging level is part of BIND's status.
rndc status | grep debug
debug level: 0
17
Set debug level to ten.
Increase the debugging level by one (now 11).
Either command will disable debugging.
18. © Men & Mice http://menandmice.com
rndc trace
> rndc trace 15
> dig +noall slaveme.dnslab.org soa
> cat named.run
25-Nov-2015 09:56:11.012 debug level is now 15
25-Nov-2015 09:56:34.167 client 192.168.53.251#49360: UDP request
25-Nov-2015 09:56:34.167 client 192.168.53.251#49360: using view '_default'
25-Nov-2015 09:56:34.167 client 192.168.53.251#49360: request is not signed
25-Nov-2015 09:56:34.167 client 192.168.53.251#49360: recursion not available
25-Nov-2015 09:56:34.167 client 192.168.53.251#49360: query
25-Nov-2015 09:56:34.167 client 192.168.53.251#49360 (slaveme.dnslab.org): ns_client_attach: ref = 1
25-Nov-2015 09:56:34.167 client 192.168.53.251#49360 (slaveme.dnslab.org): query 'slaveme.dnslab.org/
SOA/IN' approved
25-Nov-2015 09:56:34.167 client 192.168.53.251#49360 (slaveme.dnslab.org): send
25-Nov-2015 09:56:34.167 client 192.168.53.251#49360 (slaveme.dnslab.org): sendto
25-Nov-2015 09:56:34.167 client 192.168.53.251#49360 (slaveme.dnslab.org): senddone
25-Nov-2015 09:56:34.167 client 192.168.53.251#49360 (slaveme.dnslab.org): next
25-Nov-2015 09:56:34.168 client 192.168.53.251#49360 (slaveme.dnslab.org): ns_client_detach: ref = 0
25-Nov-2015 09:56:34.168 client 192.168.53.251#49360 (slaveme.dnslab.org): endrequest
25-Nov-2015 09:56:34.168 client @0xb7430868: udprecv
[...]
18
The commands were run on the
authoritative server for the
queried zone.
19. © Men & Mice http://menandmice.com
rndc trace
> rndc trace 15
> dig +noall slaveme.dnslab.org soa
> cat named.run
25-Nov-2015 09:56:11.012 debug level is now 15
25-Nov-2015 09:56:34.167 client 192.168.53.251#49360: UDP request
25-Nov-2015 09:56:34.167 client 192.168.53.251#49360: using view '_default'
25-Nov-2015 09:56:34.167 client 192.168.53.251#49360: request is not signed
25-Nov-2015 09:56:34.167 client 192.168.53.251#49360: recursion not available
25-Nov-2015 09:56:34.167 client 192.168.53.251#49360: query
25-Nov-2015 09:56:34.167 client 192.168.53.251#49360 (slaveme.dnslab.org): ns_client_attach: ref = 1
25-Nov-2015 09:56:34.167 client 192.168.53.251#49360 (slaveme.dnslab.org): query 'slaveme.dnslab.org/
SOA/IN' approved
25-Nov-2015 09:56:34.167 client 192.168.53.251#49360 (slaveme.dnslab.org): send
25-Nov-2015 09:56:34.167 client 192.168.53.251#49360 (slaveme.dnslab.org): sendto
25-Nov-2015 09:56:34.167 client 192.168.53.251#49360 (slaveme.dnslab.org): senddone
25-Nov-2015 09:56:34.167 client 192.168.53.251#49360 (slaveme.dnslab.org): next
25-Nov-2015 09:56:34.168 client 192.168.53.251#49360 (slaveme.dnslab.org): ns_client_detach: ref = 0
25-Nov-2015 09:56:34.168 client 192.168.53.251#49360 (slaveme.dnslab.org): endrequest
25-Nov-2015 09:56:34.168 client @0xb7430868: udprecv
[...]
19
original query name. use to
group all messages for one
query, on a busy server
20. © Men & Mice http://menandmice.com
BIND 9 log configuration
templates
20
21. © Men & Mice http://menandmice.com
Resolver Logging Template
Below is a template for the BIND 9 logging configuration for a DNS resolver:
logging {
channel named { file "named.log" versions 10 size 20M; print-time yes; print-category yes; };
channel security { file "security.log" versions 10 size 20M; print-time yes; };
channel dnssec { file "dnssec.log" versions 10 size 20M; print-time yes; };
channel resolver { file "resolver.log" versions 10 size 20M; print-time yes; };
channel query_log { file "query.log" versions 10 size 20M; severity debug; print-time yes; };
channel query-error { file "query-errors.log" versions 10 size 20M; severity info; print-time yes; };
category default { default_syslog; named; };
category general { default_syslog; named; };
category security { security; };
category queries { query_log; };
category dnssec { dnssec; };
category edns-disabled { default_syslog; };
category config { default_syslog; named; };
category resolver { resolver; };
category spill { default_syslog; named; };
category client { default_syslog; named; };
category network { default_syslog; named; };
category lame-servers { null; };
};
21
22. © Men & Mice http://menandmice.com
Authoritative-Server Logging Template
Below is a template for the BIND 9 logging configuration for an authoritative DNS-Server:
logging {
channel named { file "named.log" versions 10 size 20M; print-time yes; print-category yes; };
channel security { file "security.log" versions 10 size 20M; print-time yes; };
channel ratelimit { file "ratelimit.log" versions 10 size 20M; print-time yes; };
channel query_log { file "query.log" versions 10 size 20M; severity debug; print-time yes; };
channel query-error { file "query-errors.log" versions 10 size 20M; severity info; print-time yes; };
channel xfer { file "transfer.log" versions 10 size 10M; print-time yes; };
channel update { file "update.log" versions 10 size 10M; print-time yes; };
category default { default_syslog; named; };
category general { default_syslog; named; };
category security { security; };
category queries { query_log; };
category config { default_syslog; named; };
category resolver { resolver; };
category network { default_syslog; named; };
category rate-limit { ratelimit; };
category xfer-in { default_syslog; xfer; };
category xfer-out { default_syslog; xfer; };
category notify { default_syslog; xfer; };
category update { default_syslog; update; };
category update-security { default_syslog; update; };
};
22
23. © Men & Mice http://menandmice.com
Men & Mice logeater tools
23
24. © Men & Mice http://menandmice.com
Men & Mice "logeater" tools
the logeater tools are a collection of tools around BIND 9 log
files
The source code (golang) can be found at
https://github.com/menandmice-services/logeater
Binary executable for Linux (x86_64/i386), MacOS X
(x86_64) and Windows can be found on the Men & Mice
Packages download server
http://packages.menandmice.com/logeater/
24
25. © Men & Mice http://menandmice.com
Men & Mice "logeater" tools
logeater-queries will read a BIND 9 (9.10+) style query log
file and will output statistics and aggregated information
from the file:
$ ./logeater-queries -h
Usage of ./logeater-queries:
-c list query network classes
-d list domain names
-i list query IP addresses
-n no reverse IP lookup
-s list statistics
-t list query type
25
26. © Men & Mice http://menandmice.com
Men & Mice "logeater" tools
logeater-queries Example - printing overall statistics:
$ cat query.log | ./logeater-queries -s | column -t -s ":"
Query-Statistics
21385 total queries ( 100 % )
20471 iterative queries ( 95 % )
914 recursive queries ( 4 % )
863 queries over TCP ( 4 % )
16987 queries with EDNS support ( 79 % )
15197 queries indicate DNSSEC support ( 71 % )
8804 queries with DNSSEC validation disabled (CD-flag) ( 41 % )
1571 queries TSIG signed ( 7 % )
26
27. © Men & Mice http://menandmice.com
Men & Mice "logeater" tools
logeater-queries Example - printing network classes and query types of queries:
$ cat query.log | ./logeater-queries -c -t | column -t
Query-Network-Classes
21379 : IN
6 : CH
Query-Network-Types
8514 : A
4060 : AAAA
3079 : SOA
2372 : DNSKEY
927 : PTR
658 : MX
543 : NS
312 : DS
286 : TXT
186 : NSEC
129 : ANY
115 : CNAME
[…]
27
28. © Men & Mice http://menandmice.com
Men & Mice "logeater" tools
logeater-queries Example - printing the top ten query IP addresses
with reverse name resolution (can be slow):
$ cat query.log | ./logeater-queries -i | head | column -t
Query-IP-Addresses
1571 : 212.114.206.217 [muc.example.de.]
821 : 2620:74:13:4400::41 [dnsviz-db.verisignlabs.com.]
794 : 72.13.58.112 [dnsviz-db.verisignlabs.com.]
704 : 54.234.42.241 [241.compute-1.amazonaws.com.]
682 : 2001:19f0:5001:df:76d7:5703:ba0a:e220 []
565 : 185.92.221.212 [185.92.221.212.vultr.com.]
467 : 185.22.143.29 [b9168f1d.cgn.dg-w.de.]
314 : 91.51.184.46 [3b82e.dip0.t-ipconnect.de.]
28
29. © Men & Mice http://menandmice.com
Men & Mice "logeater" tools
logeater-dnssec analyses the a log file with messages from the
"DNSSEC" category and groups the error messages :
$ cat dnssec.log | ./logeater-dnssec | head
8727 : 0C9F6LGOE6NADAS8KG1CLIK9UO9G7EIG.ad/NSEC3: no valid signature found
6953 : ad/SOA: no valid signature found
3976 : sat-tv.com/A: got insecure response; parent indicates it should be secure
1730 : mozilla.com/SOA: no valid signature found
1586 : stream.bestvideostreaming.is/A: no valid signature found
1577 : 8FC1DQ3C2Q3ERFD4UO40ENDBTSFME5JO5.ad/NSEC3: no valid signature found
1576 : sat-tv.com/SOA: got insecure response; parent indicates it should be secure
1576 : cdws.eu-west-1.amazonaws.com.Cisco/AAAA: bad cache hit (amazonaws.com.Cisco/DS)
1483 : 0c9f6lgoe6n13ad9iu1clik9uo9g7eig.ad/NSEC3: no valid signature found
968 : cbr.de/NSEC: no valid signature found
29
30. © Men & Mice http://menandmice.com
Men & Mice "logeater" tools
logeater-resolver analyses the a log file with messages from the "resolver" category
and groups the error messages :
$ cat resolvers.log | ./logeater-resolvers | head
42908 : s-cnc1.qq.com/AAAA: Name qq.com (SOA) not subdomain of zone ns-cnc1.qq.com -- invalid response
42713 : s-tel1.qq.com/AAAA: Name qq.com (SOA) not subdomain of zone ns-tel1.qq.com -- invalid response
42484 : s-os1.qq.com/AAAA: Name qq.com (SOA) not subdomain of zone ns-os1.qq.com -- invalid response
42297 : s-cmn1.qq.com/AAAA: Name qq.com (SOA) not subdomain of zone ns-cmn1.qq.com -- invalid response
20346 : mails.sonymusicfans.com/DS: invalid response
10920 : tp1.glb.nist.gov/DS: invalid response
9693 : media.netd.com.tr/AAAA for client 192.0.2.165#3347: Name netd.com.tr (SOA) not subdomain of zone
media.netd.com.tr -- invalid response
7932 : service.superc.net/AAAA for client 192.0.2.11#3073: Name superc.net (SOA) not subdomain of zone
service.superc.net — invalid response
4597 : brickleonavon.com/NS for client 192.0.2.46#3073: Name . (SOA) not subdomain of zone
brickleonavon.com -- invalid response
4474 : promo.mobile.de/AAAA for client 2001:db8:1800:88:78f9:ba4:45fe:d438#48296: Name mobile.de (SOA)
not subdomain of zone promo.mobile.de -- invalid response
30
31. © Men & Mice http://menandmice.com
Filtering logs with artificial
ignorance
31
32. © Men & Mice http://menandmice.com
Artificial Ignorance
Artificial ignorance is a concept coined by Marcus Ranum. It is used to filter
the noise away from any log-file. To apply "ai"
1.Inspect a log message. Decide if the log message is a serious issue.
A.If yes, fix the issue.
B.If no, enter the message into a filter expression that hides the
message in the future.
2.Repeat 1. until all messages are gone.
3.Inspect the filtered log every day (for example, have the log output send
my mail to your inbox). Apply steps 1a or 1b for every new message.
32
http://www.ranum.com/security/computer_security/papers/ai/
33. © Men & Mice http://menandmice.com
logtemplater
logtemplater is an open source tool that implements artificial
ignorance:
• http://www.uberadmin.com/Projects/logtemplater/
• https://github.com/rondilley/tmpltr
Installation
git clone https://github.com/rondilley/tmpltr.git
cd tmpltr
./bootstrap
autoreconf -i
./configure
make
make install
33
34. © Men & Mice http://menandmice.com
logtemplater
logtemplater example:
$ cat named.log | tmpltr - | sort -n | sed -e 's/%s.*||//' | tail
Opening [-] for read
121 27-Jan-2017 00:51:38.574 client @0x7a0d47b0 2001:19f0::76d7:5703:ba0a:e220#49188 (menandmice.training): transfer of
'menandmice.training/IN': IXFR ended
121 27-Jan-2017 00:51:38.415 zone menandmice.training/IN: sending notifies (serial 315)
170 26-Jan-2017 23:00:37.060 client @0x7bf057b0 192.0.2.197#33796 (1.1.1.1.in-addr.arpa): query failed (REFUSED) for 1.1.1.1.in-
addr.arpa/IN/PTR at query.c:6899
216 26-Jan-2017 22:57:22.482 zone _openpgpkey.example.de/IN: next key event: 26-Jan-2017 23:57:22.480
216 26-Jan-2017 22:57:22.480 zone _openpgpkey.example.de/IN: reconfiguring zone keys
5184 26-Jan-2017 22:57:21.702 zone 6.b.2.0.8.9.1.0.1.0.a.2.ip6.arpa/IN: next key event: 26-Jan-2017 23:57:21.675
5184 26-Jan-2017 22:57:21.675 zone 6.b.2.0.8.9.1.0.1.0.a.2.ip6.arpa/IN: reconfiguring zone keys
6495 26-Jan-2017 22:11:20.136 client @0x7bf057b0 192.0.2.217#47130/key dyn.example.de: signer "dyn.example.de" approved
6495 26-Jan-2017 22:11:20.137 client @0x7bf057b0 192.0.2.217#47130/key dyn.example.de: updating […]
6496 26-Jan-2017 22:10:20.765 client @0x7bf057b0 192.0.2.217#42821/key dyn.example.de: updating […]
34
Date and time of 1st
occurrence of this type of
message
Number of
messages of this type
inside the file
Content of 1st occurrence of
this type of message
35. © Men & Mice http://menandmice.com
logtemplater
to filter out well-known but not important log messages, save the
template into an "ignore" file and edit the file
$ cat named.log | tmpltr w named.log.ignore —
$ $EDITOR named.log.ignore
next time, the template will be called with the "ignore" file:
$ cat named.log | tmpltr -t named.log.ignore — | sort -n | sed -e 's/%s.*||//'
from a crontab file or cron.daily script:
$ cat named.log | tmpltr - | sort -n | sed -e 's/%s.*||//' |
mailx -s 'BIND 9 Log-Message filtered' admin@example.com
35
36. © Men & Mice http://menandmice.com
Men & Mice DNS Training
•Introduction to DNS & BIND Hands-On Class
•March 6 – 8, 2017, Amsterdam NL
•April 3 – 5, 2017, Redwood City (CA), USA
•May 1 – 3, 2017, Boston (MA), USA
•June 5 – 7, 2017, Zürich, Switzerland
36
https://www.menandmice.com/support-training/training/
37. © Men & Mice http://menandmice.com
Men & Mice DNS Training
•Introduction & Advanced DNS and BIND Topics
Hands-On Class
•March 6 – 10, 2017, Amsterdam NL
•April 3 – 7, 2017, Redwood City (CA), USA
•May 1 – 5, 2017, Boston (MA), USA
•June 5 – 9, 2017, Zürich, Switzerland
37
https://www.menandmice.com/support-training/training/
38. © Men & Mice http://menandmice.com
Men & Mice DNS Training
•DNS & BIND (German Language)
•February 6 – 8, 2017, Essen, DE
•DNS Sicherheit (German Language)
•February 9-10, 2017, Essen, DE
38
http://linuxhotel.de/
39. © Men & Mice http://menandmice.com
our next webinar
DNSSEC Zone Signing Tutorial
Love it or hate it, DNSSEC can make a vital difference to protecting
your DNS. Yet starting with DNSSEC can be intimidating.
This webinar is designed as an easy-to-follow tutorial on DNSSEC
signing a zone for DNS admins. Our focus will be on DNSSEC
zone signing automation with the Knot DNS Server and BIND 9.
Join us for a 45-minute webinar with a Q&A session at the end, on
Thursday, March 23, 2017
at 4:00 PM CET/ 3:00 PM GMT/ 11:00 AM EDT / 8:00 AM PDT.
39
https://www.menandmice.com/resources/educational-resources/webinars/dnssec-zone-signing-tutorial/
40. © Men & Mice http://menandmice,com
Thank you!
Questions? Comments?
40