This document summarizes the architecture of Segment's analytics event routing system. It discusses constraints of high throughput and reliability. The initial system used RabbitMQ and MongoDB but scaled to NSQ and Go microservices. Metrics and queues were used to monitor and schedule fan-out of events. Microservices provided isolation and visibility while Docker enabled easy deployment. Looking ahead, the system may move to Kafka and standardize the microservices toolkit.

1. Routing billions of
analytics events
with high
deliverability
Calvin French-Owen
@calvinfo

2. This talk
- Constraints
- Architecture
- Monitoring and microservices

3. What problem are
we solving?

8. - Thousands of incoming requests/second
- Need to for real-time
- Reliable fan-out
- Hundreds of unreliable APIs
Our Constraints

9. - Delivery rate
- End-to-end latency
- Data fidelity
Golden Metrics

10. The Lifecycle of an
Event

12. 3 months post
launch: 80 req/s

13. Segment V1
- Node everywhere
- RabbitMQ
- EC2/VPC
- Mongo
- Redis

15. API
Layer

16. 3 months post
launch: 80 req/s

18. API Scaling
- Rabbit → NSQ
- Node → Go
- Making the edge stateless

19. - Removed AMQP (segmentio/nsq.js)
- Co-located
- Distributed
- Simple and rock solid
Rabbit → NSQ

24. But there were still
issues...

28. // parse the body
var body = JSON.parse(req.body);
// elsewhere...
clone(body);

29. // parse the body
var body = JSON.parse(req.body);
// elsewhere...
clone(body); //

31. stateless!!!

32. stateless!!!

33. - Edge nodes are stateless
- NSQ for simplicity and reliability
- Go for parallelism
Scaling the API

36. Ingestion and
fanout

37. - Queueing topology
- Abstract everything
- Sanely retry
Scaling the fanout

38. Queues give you
flexibility and
scheduling

39. Queues give you
flexibility and
scheduling

47. If you’re building
150 of anything, err
on the side of over-
abstraction

49. // Integration Factory
function createIntegration(name){
// Create the constructor to be passed back
function Integration(settings){
this.debug = debug('segmentio:integration:' + this.slug());
this.settings = settings;
this.initialize();
}
Integration.prototype.name = name; // set the name
merge(Integration.prototype, proto); // add prototype methods
merge(Integration, statics); // add static methods
return Integration; // return the constructor
}

50. var MailChimp = module.exports = integration('MailChimp')
.channels(['server', 'mobile', 'client'])
.endpoint('https://api.mailchimp.com/')
.ensure('settings.datacenter')
.ensure('settings.apiKey')
.ensure('settings.listId', { methods: ['identify'] })
.ensure('message.email')
.mapper(mapper) // map our input to our output
.retries(2);

51. Integration.prototype.track = function track(payload, fn){
var self = this;
return this
.get('/httpapi') // common request handling
.type('json')
.query({ api_key: this.settings.apiKey })
.query({ event: JSON.stringify(payload) })
.end(function(err, res){
if (err) return fn(err, res);
if ('invalid api_key' == res.text) return fn(self.error('invalid api_key'));
fn(null, res);
});
}

52. Retries
function status(err){
return err.status == 500
|| err.status == 502
|| err.status == 503
|| err.status == 504
|| err.status == 429;
}
function network(err){
return err.code == 'ECONNRESET'
|| err.code == 'ECONNREFUSED'
|| err.code == 'ECONNABORTED'
|| err.code == 'ETIMEDOUT'
|| err.code == 'EADDRINFO'
|| err.code == 'EHOSTUNREACH'
|| err.code == 'ENOTFOUND';
}
API Errors Network Errors

54. // retry strategy with exponential backoff
if (err.retry) {
var attempts = msg.attempts;
var timeout = jitter(15*Math.pow(attempts, 3));
msg.requeue(timeout);
return;
}

55. Microservices &
Monitoring

56. - Microservices everywhere
- Docker for isolation
- Use metrics religiously
Microservices & Monitoring

59. module "google-analytics" {
source = "./worker"
cluster = "integration-worker"
memory = "256"
cpu = "128"
name = "google-analytics"
version = "latest"
count = "${var.count}"
}

62. The more surface
area you have, the
more visibility you
need.

67. Scaling your data pipeline
1. Queues not only define service boundaries, but
scheduling
2. Microservices and workers can provide great visibility
and scalability–as long as they are easy to boot
3. The bigger your surface area, the more visibility and
metrics you will need to provide

68. What’s next?

69. - In search of fairness
- Moving to Kafka
- Standard microservice toolkit
- Custom data transforms
What’s next?

70. Fin
Questions?
calvin@segment.com
@calvinfo