Distributed and massively scalable systems are difficult to design, implement, and operate. Further, microservice architectures are supposed to enable your business to be disruptive and innovative. Cloud Foundry provides that solution for modern, cloud native architectures and accelerates the ability to transform operational and development practices. The aggregated logging subsystem of Cloud Foundry, the Loggregator, is a key component of this transformation. And within the Loggregator, the firehose is where log and metric data is consumed. This talk will walk through how to consume the firehose and dive into the anatomy of the various firehose message types. Several use cases and lessons learned monitoring the platform versus an application will be covered to further demonstrate how Cloud Foundry can facilitate innovation.

1. Monitoring Cloud Foundry:
Learning about the Firehose
Dustin Ruehle
Cloud Architect, ECS Team
@dustinruehle, druehle@ecsteam.com
Tom Collings
Cloud Architect, ECS Team
tcollings@ecsteam.comWednesday May 25, 2016

2. Welcome to Cloud foundry
Cloud Native Developers

3. Welcome to Cloud foundry
Cloud Native Developers
Apps

4. Welcome to Cloud foundry
CONTINUOUS DELIVERY PIPELINE (TESTING, BUILD AND RELEASE AUTOMATION)
NON-PROD CLOUD FOUNDRY PROD CLOUD FOUNDRY
Cloud Native Developers
Apps
CI/CD Pipeline

5. Does it feel more like….

6. Does it feel more like….Welcome to the Jungle
What’s the health of my platform?
How do I get statistics on
what’s been happening?
How are my apps doing?

7. The “Loggregator” to the rescue

8. How do metrics and logs get to the firehose…
Sources

9. How do metrics and logs get to the firehose…
Sources Metron
Agents

10. How do metrics and logs get to the firehose…
Sources Metron
Agents
Dopplers

11. How do metrics and logs get to the firehose…
Sources Metron
Agents
Dopplers Traffic
Controllers

12. There it is…. the firehose
Sources Metron
Agents
Dopplers
firehose
Traffic
Controllers

13. Make your choice…
Sources Metron
Agents
Dopplers
firehose
Traffic
Controllers

14. Firehose – just a multiplexer
Sources Metron
Agents
Dopplers
firehose
Traffic
Controllers
dropsonde

15. What are the other ”logging points”?
Job
syslog
System
Events
App
drain
App
Events
Syslog Aggregator
(Splunk, ELK)
firehose
Loggregator Roadmap

16. What are the other ”logging points”?
Job
syslog
System
Events
App
drain
App
Events
Syslog Aggregator
(Splunk, ELK)
cf logs myapp
firehose

17. Firehose - Let’s take a look…. ZERO app activity!

18. Firehose - Did that feel like…

19. Firehose - How much was that?
Cloud Foundry
bosh-lite
19 “VMs” 60 seconds
firehose

20. Firehose - How much was that?
Cloud Foundry
bosh-lite
19 “VMs”
firehose

21. How do I consume this firehose?
Cloud Foundry
firehose

22. How do I consume this firehose?
Cloud Foundry
firehose

23. Required for connecting to the firehose
Cloud Foundry
bosh-lite
firehose
websocket
endpoint
Requires user with
firehose.doppler scope
doppler endpoint
wss://doppler.bosh-lite.com:4443

24. Required for connecting to the firehose
Cloud Foundry
bosh-lite
firehose
websocket
endpoint
Requires user with
firehose.doppler scope
doppler endpoint
wss://doppler.bosh-lite.com:4443
discoverable via api
/v2/info

25. What now?....events
Cloud Foundry
firehose
dropsonde
• Envelopes containing events
• Envelopes come from the
traffic controller

26. What now?....events
Cloud Foundry
firehose
dropsonde
Unmarshall from protocol buffers

27. What now?....events
Cloud Foundry
firehose
dropsonde
• Event types
• HttpStart
• HttpStop
• HttpStartStop
• LogMessage
• ValueMetric
• CounterEvent
• Error
• ContainerMetric
More details - https://github.com/cloudfoundry/dropsonde-protocol/blob/master/events/README.md

28. Nozzle lifecycle options
Cloud Foundry
firehose
bosh deployment
cf app
cf plugin

29. Nozzle examples
bosh deployment
cf app
cf plugin
https://docs.cloudfoundry.org/loggregator/nozzle-tutorial.html
NOAA nozzle – https://github.com/cloudfoundry/noaa
Firehose-to-syslog –
https://github.com/cloudfoundry-community/firehose-to-syslog
Firehose plugin– https://github.com/cloudfoundry/firehose-plugin

30. Let’s look at an example message or two….
cf logs spring-music
cf nozzle --filter LogMessage

31. One of these things is not like the other…
cf logs spring-music
cf nozzle --filter LogMessage

32. Here is an app log message…
cf logs spring-music
cf nozzle --filter LogMessage
App guid in the firehose

33. Here is an app log message…
cf logs spring-music
cf nozzle --filter LogMessage
What happens when doing blue-green deploys?

34. Here is an app log message…
cf logs spring-music
cf nozzle --filter LogMessage
What happens when doing blue-green deploys?
Resolved using firehose-to-syslog nozzle
Injects the app name into the nozzle output

35. A novel use case – feedback control loops

36. Feedback control demo – Memory based auto-scaling
memory-based-autoscaler nozzle (cf app)
firehose
Cloud
Foundry
bosh-lite

37. Feedback control demo – Memory based auto-scaling
memory-based-autoscaler nozzle (cf app)
firehose
Cloud
Foundry
bosh-lite ContainerMetric events
for dummy app

38. Feedback control demo – Memory based auto-scaling
memory-based-autoscaler nozzle (cf app)
firehose
Cloud
Foundry
bosh-lite Average memory for all
instances exceeds a
threshold for specified
duration

39. Feedback control demo – Memory based auto-scaling
memory-based-autoscaler nozzle (cf app)
firehose
Cloud
Foundry
bosh-lite cf scale using the cf API

40. Demo

41. Where can I get “the code”?
Memory auto-scaler
https://github.com/ECSTeam/memory-based-autoscaler
Memory user
https://github.com/ECSTeam/memory-user

42. For more information….
•Come by the ECS Team booth – we’ll talk firehose madness
•Register to win
• Amazon Echo
• Garmin Vivosmart
•Drawing to win – 3:20pm today (5/25/2016)