Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Staying lean with application logs


Published on

When you're starting out, staying nimble is paramount. This talk will help you push back infrastructure decisions and keep your options open by storing machine-readable JSON logs, not relational data, and processing them as an event stream to construct your data model on the fly.

Published in: Software
  • Be the first to comment

  • Be the first to like this

Staying lean with application logs

  1. 1. Staying Lean With Application Logs @SamirTalwar and @sleepyfox 2016-09-22T11:10:00+0200 Essay form at
  2. 2. Reactive,
 Event-Sourced, and Lean
 (also with logs) @SamirTalwar and @sleepyfox 2016-09-22T11:10:00+0200 Essay form at
  3. 3. We're making a game for Xbox Two.
  4. 4. Trials of Anubis One of the most popular game modes. You have two teams, Alpha and Bravo, each of three players. Each team is trying to defeat the other one. If you take another player down, they don’t revive in the same round. Players can bring their teammates back up, but it’s time- consuming. If all the players on the team are taken down, that team loses. Best out of three rounds wins the match.
  5. 5. Let's get down to business. I'm here today to talk about infrastructure, not graphics.
  6. 6. Phase One
  7. 7. Phase One This is a monolith. It's easy to reason about. But it doesn't scale past a few games.
  8. 8. Phase Two
  9. 9. Phase Two Deploy many monoliths. It's still easy to reason about. The components are unbalanced. They still have to share data.
  10. 10. Phase Three
  11. 11. Phase Three We've split into services. Because microservices are a thing now. Communication via the database went well… for about 20 seconds, before we hit consistency issues. We're bottlenecked on the database.
  12. 12. Phase Four
  13. 13. Phase Four Services manage their own data storage. We do communication over HTTP, because that's apparently lightweight now.
  14. 14. Phase Four Four out of six service components are just managing HTTP. Every service needs the same kind of components. None of the HTTP stuff makes us money. It’s just waste.
  15. 15. We can do better. Let's make something that works with our architecture instead.
  16. 16. Event Sourcing What if, rather than consuming from other services, each service published every single event? • no HTTP servers • no HTTP logs; just publish to a store • still need the clients, but they're simpler
  17. 17. Event Sourcing Sounds like a solution. Just one problem. We'd have to rewrite everything.
  18. 18. Event Sourcing Wait. We already publish events. To STDOUT. 😮 We log everything already. Push them to Fluentd or Logstash.
  19. 19. A log-powered event bus. It'll never work. Let's do it.
  20. 20. impressive
 goes here
  21. 21. What's in a log? Machine-readable first. Human-readable second. If you want to read your logs,
 process them first. Kibana will help a lot.
  22. 22. What's in a log? We don't care about: • filtering by severity, • log rotation, • multiple formats, • or even logging to a file. Those are ops concerns, not dev.
  23. 23. What's in a log? Every event has a type. The structure of the event is determined by this type. That structure is a public contract… Just like a Java class. Versioning is probably required.
  24. 24. ygt/microservice-logging const Logger = require('./logger')
 const log = new Logger({
 output: console,
 events: {
 startup: 'startup',
 httpRequest: 'HTTP request',
 database: 'database'
 }).with({service: 'my super service'})
  25. 25. ygt/microservice-logging{
 message: 'Ready to go.',
 port: 8080
 // later on
 const requestLog = log.with({
 request_id: '126bb6fa-28a2-470f-b013-eefbf9182b2d'})
 message: 'Connection failed.'}){
 request: {method: 'GET'},
 response: {status: 500}})
  26. 26. ygt/microservice-logging {"timestamp":1474443152917,
 "service":"my super service",
 "message":"Ready to go.",
 "port":8080} {"timestamp":1474443152920,
 "service":"my super service",
 "message":"Connection failed."} {"timestamp":1474443152921,
 "event_type":"HTTP request",
 "service":"my super service",
  27. 27. ygt/microservice-logging Check it out. npm install microservice-logging
  28. 28. So. We have a log of all events in the system. There’s real-time reactive behaviour across services. The stateless services are completely scalable… … and there’s patterns for scaling the stateful ones. Adding new services that derive information from the data requires no modifications to the existing ones. We have a model for handling catastrophic failure. Reporting and analytics are practically free now.
  29. 29. So. And my favourite:
 we don't have a database at all.
  30. 30. Scaling out Load-balance, with sticky sessions based on IDs if services keep state in memory. Avoid in-memory state; outsource to Memcached or Redis. On failure, respawn and replay. Push events from Fluentd/Logstash to Kafka or Kinesis. Flag non-JSON logs for manual review. Store events in Elasticsearch or Cassandra. When you need a real database, keep your events, and populate it from there (write model and read model).
  31. 31. Log everything Who's got questions?
  32. 32. Thank you. @SamirTalwar