Presented at the Auckland AWS Meet-up: In this meet-up, Chris will take us through an interactive session that will examine log solutions in the cloud. We'll take a look at some possible build-your-own architectures on AWS, common tools and practices, and commercial options. We'll then demo logging data from an EC2 Instance using Amazon Kinesis, Amazon Elasticsearch Service and S3.

1. Logging @ Scale on AWS
Visualization > Plaintext

2. Who am I?
• Chris Riddell
• Tech Co-Founder of some start ups, Senior Software Engineer
• Big Data guy – Redshift, S3, EC2, EMR, Hive, Spark, Dynamo and
many others….
• I manage big AWS infrastructure from software to architecture to
DevOps (start up life...)
• Java (+ others) / AWS / geek
• AWS Professionally Certified Solution Architect (ask me how!)
• Not a logging expert - like to think I’m getting close J

3. Logging @ Scale on AWS
• Use cases
• What to log
• Commercial options
• Common tools
• Plausible architectures
• Demo! EC2 w/Fluentd -> Kinesis Firehose -> Elasticsearch

4. Why log?
• Get visibility into app health, centrally accessible and
searchable
• Alerts & fast error event response – Agility!
• Not having to login to individual instances and tail
• Contextual tracking e.g. user behaviour
• Finding code optimisations
• …and many many other reasons!

5. First have common language
• What is DEBUG, INFO, WARN and ERROR used for in your
organisation?
• Have common language for what should be logged where
• Bad leveling messes up your storage
• E.g. DEBUG logs going to your expensive Elasticsearch store, when
they never need to be searched
• One guy’s opinion:
http://stackoverflow.com/a/8021604/3843660

6. Centralising the logs
• Let’s get them off the host
• Basic DIY:
• Syslog-ng, rsyslogd, nxlog
• Advanced DIY:
• Splunk forwarder, Logstash, Flume, Fluentd
• Third party
• SaaS

7. Commercial options: SaaS
• LogEntries, Sumo Logic, Loggly, Splunk Cloud, PaperTrails,
AWS CloudWatch Logs…
• Typically RESTful JSON log dump APIs
• Search & visualizations are core features
• Most have a free tier
• Many libraries available for various languages and/or
packaged versions
• Costs go up with data size, retention period and user count
• Nice to have: User defined alerts; S3 archival…..

8. Self-hosted solutions
• Splunk – Enterprise solution consisting of forwarders, search
heads, and indexers. Licensed.
• ELK/EFK stack = Elasticsearch, Logstash or Fluentd, Kibana
• Today = Fluentd (forwarder), Kinesis Firehose, Elasticsearch,
Kibana
• FKFEK stack?

9. Elasticsearch?
• For search!
• Indexes
• Shards - Distributed & scales out
• Replicas
• JSON REST API
• Apache Lucene
• Kibana is an Elasticsearch plugin that provides a nice
interface to the search data *with visualizations*

10. Logstash & Fluentd agents
• Packaged install
• Input and output logs
• Centralise your instance logs
• Often used as a syslog tail’er or as a local HTTP log endpoint
• Parse/transform/filter/tag
• Store or Forward
• “Logstash emphasizes flexibility and
interoperability whereas Fluentd prioritizes simplicity and
robustness” - http://goo.gl/f5I4cL

11. Lo

12. Architecture – Pre October 2015

13. Today’s Solution

14. Lo
Syslog/Fluentd
on EC2
Kinesis
Firehose Elasticsearch

15. Demo: Set up EC2 and Fluentd
• We spin up a default AWS AMI EC2 instance with role
permission to push data to Firehose, access via SSH & HTTP)
• We SSH in and install Fluentd
• curl -L https://td-
toolbelt.herokuapp.com/sh/install-redhat-td-
agent2.sh | sh
• /usr/sbin/td-agent-gem install fluent-plugin-
kinesis #install AWS FH plugin
• Then configure fluentd to push our syslog’s

16. Demo: Fluentd config (/etc/td-agent/td-
agent.conf)
## Syslog reader. Configure port 42185 to send events to in rsyslog config
<source>
type syslog
port 42185
bind 0.0.0.0
tag system
</source>
## Filters to transform records and add metadata
<filter **>
type record_transformer
enable_ruby
<record>
@timestamp ${require 'time'; Time.now.utc.iso8601}
</record>
</filter>
## Output to Firehose using the instance role
<match **>
@type kinesis_firehose
region us-west-2
delivery_stream_name logs
flush_interval 2s
</match>

17. Demo: restart log agents and serve HTTP
# After /etc/td-agent/td-agent.conf has been setup
# Send syslog to fluentd listener
echo "*.* @127.0.0.1:42185" | sudo tee /etc/rsyslog.d/22-
fluent.conf
sudo service td-agent restart
sudo service rsyslog restart
# Let’s make a web server for you to push your own logs!
mkdir web
cd web
echo 'Hello!' > index.html
sudo python -m SimpleHTTPServer 80 |& logger -t httpsvr &

18. Demo: Setting up Elasticsearch
• We set up AWS Elasticsearch Service: Some notes:
• Dedicated master - performs cluster management tasks, doesn’t
hold data
• Metrics: The usual stuff. Note the JVMMemoryPressuremetric.
Amazon recommends scale up/out if > 85%
• Cluster status is yellow on single node because replicas cannot be
assigned. Add a node or change the setting
• Cluster has it’s own access policy. If you choose instance role
access control, you must sign all requests to ES (use AWS SDKs).
You will not be able to access Kibana on this setting
• Check what size the instance store is on your selected instance
type, or use EBS

19. Demo: Setting up Kinesis Firehose
• We create a logs delivery stream with Elasticsearch as the
target
• Firehose: Some notes:
• A pipeline to push data in at high scale
• Dump data in, and it buffers recrods(logs in our case) them before
pushing to Elasticsearch and optionally S3 (Redshift also
supported)
• Pay per GB of ingestion $0.035USD (each record rounded to
nearest 5kb)
• Different destinations (e.g. WARN/ERROR to Elasticsearch but rest
only to S3) would need different Firehose delivery streams

21. Limitations? Further features?
• AWS’s Elasticsearch is limited on plugins, but very good out of the box
settings
• Fast way to get high throughput, high scale logs into a stored index with
S3 backups, and visualisation!
• Further features using AWS Lambda as glue:
• Alerts
• CloudTrial/S3 logs ingestion to FH/Elasticsearch,
• Deletion of old indexes at a specified period (we want the last X days only)
• and more…
• S3 bucket policy for storage optimisation (eg old stuff to glacier)
• Your custom applications: Push directly to Fluentd’s HTTP endpoint, not
via syslog (more flexibility and tagging)
• Further customisations to Fluentd config

22. The end!
• Thanks!
• @ChrisJRiddell
• @ParrotAnalytics
• Hiring Intermediate /Senior Java Engineers J
• Upcoming: Web dev & more engineers
• https://parrot-analytics.workable.com/ to apply