Log data contains some of the most valuable raw information you can gather and analyze about your infrastructure and applications. Amid the mess of confusing lines of seemingly random text can be hints about performance, security, flaws in code, user access patterns, and other operational data. Without the proper tools, finding insights in these logs can be like searching for a hay-colored needle in a haystack. In this session you learn what practices and patterns you can easily implement that can help you better understand your log files. You see how you can customize web logs to add more information to them, how to digest logs from around your infrastructure, and how to analyze your log files in near real time.

2. Chris Munns -@chrismunns

3. https://secure.flickr.com/photos/psd/4389135567/

7. https://secure.flickr.com/photos/iangbl/338035861

9. client
mobile client
CloudFront
region
VPC
Amazon S3
MySQL DB
instance
Web
instances
Elastic Load
Balancing
App
instances
Elastic Load
Balancing

10. client
mobile client
CloudFront
region
VPC
Amazon S3
MySQL DB
instance
Web
instances
Elastic Load
Balancing
App
instances
Elastic Load
Balancing

11. https://secure.flickr.com/photos/hk_brian/5753530941

12. Is this one important?
https://secure.flickr.com/photos/hk_brian/5753530941

13. Is this one important? What about this one?
https://secure.flickr.com/photos/hk_brian/5753530941

15. Is this one important?

16. Is this one important? What about this one?

17. Let’s go back to the beginning
https://secure.flickr.com/photos/paukrus/9826882836

19. Numerical code
Facility
0
kernel messages
1
user-level messages
2
mail system
3
system daemons
4
security/authorization messages
…
23
local use 7 (local7)
Numerical code
Severity
0
Emergency
1
Alert
2
Critical
3
Error
4
Warning
…
7
Debug

20. <34>1 2003-10-11T22:14:15.003Z mymachine.example.comsu-ID47 -BOM'suroot' failed for lonvickon /dev/pts/8

22. Easy, right?
https://secure.flickr.com/photos/21734563@N04/2225069096

24. 66.249.64.XXX --[07/Sep/2014:08:33:43 +0000] "GET / HTTP/1.1" 200 819 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

26. Thanks for the history lesson, Chris! Now what do I do?
https://secure.flickr.com/photos/decade_null/142235888

27. *Each step has several moving pieces*
https://secure.flickr.com/photos/james_wheeler/9619984584

29. Apache LogFormat:
173.248.147.XXX --[16/Sep/2014:15:36:31 +0000] "GET / HTTP/1.1" 200 819 "-" "Pingdom.com_bot_version_1.4_(http://www.pingdom.com/)"
"%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-agent}i""

30. Customize log data
%D = The time taken to serve the request, in microseconds
%T = The time taken to serve the request, in seconds
%v = The canonical ServerName of the server serving the request
%{Foobar}C = The contents of cookie Foobarin the request sent to the server
%{Foobar}n = The contents of note Foobarfrom another module
Source: https://httpd.apache.org/docs/2.2/mod/mod_log_config.html
Apache LogFormat:

31. Apache LogFormat:
64.237.55.3 php-app1 [16/Sep/2014:16:21:31 +0000] "GET / HTTP/1.1" 200 819 23765"-" "Pingdom.com_bot_version_1.4_(http://www.pingdom.com/)"
"%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-agent}i""
"%h %v %t "%r" %>s %b %D"%{Referer}i" "%{User-agent}i""
173.248.147.XXX --[16/Sep/2014:15:36:31 +0000] "GET / HTTP/1.1" 200 819 "-" "Pingdom.com_bot_version_1.4_(http://www.pingdom.com/)"

32. Apache vs. NginxCLF patterns:
'$remote_addr-$remote_user[$time_local] ' '"$request" $status $body_bytes_sent' '"$http_referer" "$http_user_agent"'
"%h %l %u %t "%r" %>s %b "%{Referer}i" "%{User-agent}i""

35. Either way!!! Get that data off your host ASAP!!
https://secure.flickr.com/photos/foresthistory/3663382060

36. Why?

37. Instance failure. Filled disks. Auto Scaling actions.
https://secure.flickr.com/photos/eurleif/186807023

39. syslog-ng,rsyslog, nxlog
Pros:
•Open source
–Linux, Windows,and almost everything else!
•Both variants of syslogd
–Add filtering, flexible configuration, TCP as a transport
•Runs as an OS process
•Typically take the centralized data and feed into another analytics tool
•Can often accept logs from third-party sources like network devices
Central logging instance
virtual private cloud
App instances
Etc.
instances
Web instances

40. syslog-ng, rsyslog, nxlog
Cons:
•No built-in analytics/dashboard abilities
•Typically centralized host can become a single point of failure
•Potentially more difficult to scale
–Federate logs to different centralized hosts?
Central logging instance
virtual private cloud
App instances
Etc.
instances
Web instances

41. Splunk
Pros:
•Enterprise grade
•Extremely scalable
•Fault tolerance and load balancing built in
•Security of data built in
•Can technically accept data from other third-party sources as well
•Full log forwarding, analyzing, dashboardingstack + third-party apps
Splunkindexer
virtual private cloud
Splunkindexer
App instances
Etc.
instances
Web instances

42. Splunk
Cons:
•Enterprise-grade pricing
•Enterprise-grade licensing
•Indexer resources become an important part of capacity planning
A great option for Enterprises and large shops!
Splunkindexer
virtual private cloud
Splunkindexer
App instances
Etc.
instances
Web instances

43. Logstash
Pros:
• Open source
• Extremely scalable
• Fault tolerance built in
• Support offerings from Elasticsearch!
• Active code base and ecosystem
• Pluggable
• Ties in with other tools for
dashboarding/analytics
virtual private cloud
App
instances
Etc.
instances
Web
instances
Redis
Elasticsearch Elasticsearch Elasticsearch
Logstash
indexer
Logstash
indexer

44. Logstash
Cons:
• “ELK Stack” has many moving pieces
• Lot of DIY to getting it set up
• Very quickly changing/improving
technology stack
Most popular open source option today!
virtual private cloud
App
instances
Etc.
instances
Web
instances
Redis
Elasticsearch Elasticsearch Elasticsearch
Logstash
indexer
Logstash
indexer

45. SaaSoptions
Pros:
•Hosted
•Very easy to get started with
•No concerns about scaling yourself
•Flexible pricing methods
•Support
•Either their agents or syslog to them
•Built-in dashboards/analytics tools
•Constantly adding features/capabilities
virtual private cloud
NAT instance
App instances
Etc.
instances
Web instances

46. SaaSoptions
Cons:
•Data leaving your control/infrastructure
•Some restrictiveness in flexibility of the dashboards, collection agents, archive limits
SaaSmakes a lot of sense if you are small and trying to move fast and should be focusing on product first!
virtual private cloud
NAT instance
App instances
Etc.
instances
Web instances

49. Part of Amazon CloudWatchservice
CloudWatch

50. # yum install awslogs
# tail -n 7 /etc/awslogs/awslogs.conf
# awslogs put-metric-filter
--log-group-name
--filter-name
--filter-pattern
--metric-transformations
CloudWatch

54. https://github.com/etsy/logster
[root@php-app1 logster-master]# /usr/bin/logster--dry-run --output=ganglia SampleLogster/var/log/httpd/access_log
...
/usr/bin/gmetric-d 180 -c /etc/ganglia/gmond.conf--name http_2xx --value 0.533333333333 --type float --units "Responses per sec”
...

55. •Can process log files on the fly outputting metric data to numerous services:
–CloudWatch
–Ganglia
–Graphite via statsd
–Boundary
–DataDog
–many others!
•Runs as a constantly running daemon
•Little bit easier than Logster
•Can do metric output and full log centralization at the same time!
input {
file {
path => "/var/log/apache/access.log"
type => "apache-access” }
}
filter {
grok{
type => "apache-access"
pattern => "%{COMBINEDAPACHELOG}” }
}
output {
statsd{
# Count one hit every event by response
increment => "apache.response.%{response}” }
} (from: http://logstash.net/docs/1.4.2/tutorials/metrics-from-logs)
Logstash

57. Dashboards
https://secure.flickr.com/photos/joeross/6544781203

63. Dashboard for Logstash

65. Each of these examples took less than an hour to set up!

69. Focus first on what affects your customers:
Then on important technical issues:

70. Alarming:
Send alarms with:

72. Log backup& archiving
https://secure.flickr.com/photos/ant-ti/6016877003

75. How to do it right:
1.Get data into AmazonS3
2.Get data into AmazonGlacier
Amazon S3
Amazon Glacier

76. Amazon S3
Amazon Glacier

77. Sounds easy!
Amazon S3
Amazon Glacier
"MyLoggingBucket": {
"Type": "AWS::S3::Bucket",
"Properties": {
"BucketName": "MyLoggingBucket"
"LifecycleConfiguration": {
"Rules": [
{
"Id": "GlacierRule"
"Prefix": "logs",
"Status": "Enabled",
"ExpirationInDays": "365",
"Transition": {
"TransitionInDays": ”30",
"StorageClass": "Glacier"
}
}
]
}
}
}

78. Given the importance of log data, securing them properly is also important:
IAM

79. Don’t do this by hand! Make use of tools:
Build basic log centralization into every AMI!
directory "/opt/aws/cloudwatch" do
recursive true
end
remote_file"/opt/aws/cloudwatch/awslogs-agent- setup.py" do
source "https://s3.amazonaws.com/aws-cloudwatch/ downloads/latest/awslogs-agent-setup.py"
mode "0755"
end
execute "Install CloudWatchLogs agent" do
command "/opt/aws/cloudwatch/awslogs-agent- setup.py-n -r us-west-2 -c /etc/cwlogs.cfg"
not_if{ system "pgrep-f aws-logs-agent-setup" }
end

81. https://secure.flickr.com/photos/pfly/1537122018In Closing

82. Logs ARE important!

83. https://secure.flickr.com/photos/ocarchives/5333790414Logs are fun!

84. Spend the time to do log analysis right!

86. https://secure.flickr.com/photos/dullhunk/202872717/

87. Please give us your feedback on this session.
Complete session evaluations and earn re:Invent swag.
http://bit.ly/awsevals