How we built our our data plane using NATS. Unifying the logging for debugging and exporting the data of our different microservices.

1. NATS and Netlify
Building out a data plane for a globally distributed system
@ry_boflavin @netlify

2. About Me: Ryan Neal
- Head of Infrastructure at Netlify
- Simultaneously fixing and breaking everything
- Senior Dev at Yelp
- Internal tools and metrics team
- Used to about 400k metrics/sec
- 12-18k pageviews/sec
- FDE at Palantir
- Developed counter terrorist software
- 4 Billion records / day
@ry_boflavin @netlify

3. @ry_boflavin @netlify
A developer’s toolkit for deploying git-backed,
browser-driven sites to an intelligent CDN
- Global CDN
- CI cluster
- Redundant DNS
- Prerender cluster
- Mongo cluster
- Rails cluster
- 4 cloud providers
- 14 PoPs

4. API cluster
Global CDN
Pre-Render cluster
CI cluster
Distributed systems are cool
buildbot
buildbot
buildbot
buildbot
API
API
API
API
CDN CDN CDN CDN CDN CDN CDN API
API
DB land

5. The problem
your request
@ry_boflavin @netlify
CDN Node
db

6. The problem
your request
@ry_boflavin @netlify
CDN Node
db

7. The problem
your request
@ry_boflavin @netlify
CDN Node
proxy
db

8. The problem
your request
@ry_boflavin @netlify
CDN Node
proxy origin
db

9. The problem
your request
@ry_boflavin @netlify
CDN Node
proxy origin
api
db

10. The problem
your request
@ry_boflavin @netlify
CDN Node
db
X
X
X

11. Unity
- Cohesive view of system
- Traceability between services
- Build for now, not later

12. The Naive Solution
@ry_boflavin @netlify
random service
logs
Papertrail
daemon
papertrailrandom service

13. Immediate Problem
- Make the logs searchable
- Easy to add more logs
Long Term Vision
- A generic system to let services push data out
- An easy way to access that data for new and fun uses
Tool Requirements
- Easy installation
- Good scaling factors
- Secure
Spec before building
@ry_boflavin @netlify

14. And so the story begins...
@ry_boflavin @netlify
Rabbit MQ
- Existing infrastructure
- Didn’t need enterprise messaging features
- Data was only metrics, telemetry and logs
Kafka
- Didn’t want to run zookeeper
- Didn’t need rewind or buffering

15. Creating the Data plane
@ry_boflavin @netlify
logs nats
random service

16. Creating the Data plane
@ry_boflavin @netlify
logs nats
random service
streamer

17. Creating the Data plane
@ry_boflavin @netlify
random service
logs nats
random service
streamer

18. Creating the Data plane
@ry_boflavin @netlify
random service
logs nats
random service
streamer elastinats
es

19. Creating the Data plane
@ry_boflavin @netlify
random service
logs nats
random service
streamer
elastinats
elastinats
elastinats
elastinats
es

20. Creating the Data plane
@ry_boflavin @netlify
random service
logs nats
random service
streamer
taptap
elastinats
elastinats
elastinats
elastinats
es

21. Elastinats lessons
@ry_boflavin @netlify
func(m *nats.Msg) {
stats.IncrementMessagesConsumed()
go func() {
payload := message.NewPayload(string(m.Data), m.Subject)
// maybe it is json!
_ = json.Unmarshal(m.Data, payload)
c <- payload
}()
}
func(m *nats.Msg) {
stats.IncrementMessagesConsumed()
payload := message.NewPayload(string(m.Data), m.Subject)
// maybe it is json!
_ = json.Unmarshal(m.Data, payload)
c <- payload
}
- Don’t block the consumer

22. Elastinats lessons
@ry_boflavin @netlify
- Don’t block the consumer
- Use ES’s Bulk API

23. Elastinats lessons
@ry_boflavin @netlify
- Don’t block the consumer
- Use ES’s Bulk API
- Add error reporting
handle := func(nc *nats.Conn, sub *nats.Subscription, err error) {
log.Warn(err)
}
nc, err := nats.Connect(serverString, nats.Secure(tlsConfig), nats.ErrorHandler(handle))
if err != nil {
panic(err)
}

24. Elastinats lessons
@ry_boflavin @netlify
- Don’t block the consumer
- Use ES’s Bulk API
- Add error reporting
- Use buffering
ch := make(chan *nats.Msg, 100000)
sub, err := nc.ChanSubscribe(subject, ch)
if err != nil {
log.Fatal("Failed to subscribe")
}
defer sub.Unsubscribe()
sub, err := nc.SubscribeSync(subject)
if err != nil {
log.Fatal("Failed to subscribe")
}
defer sub.Unsubscribe()
err := sub.SetPendingLimits(numMsgs, numBytes)

25. Future Work
@ry_boflavin @netlify

26. Future Work
@ry_boflavin @netlify
- Use a nats_metrics library to measure and push to nats
- Add more taps for log analysis
- Migrate legacy services to push based metrics and logs

27. @ry_boflavin @netlify
Questions?

28. Links
https://github.com/netlify/elastinats
https://github.com/netlify/streamer
https://github.com/rybit/nats_metrics
@ry_boflavin @netlify
https://github.com/rybit
ryan@netlify.com
Check out the slides on slideshare!!