Fake It 'Til You Make It
•Download as PPTX, PDF•
0 likes•109 views
Presented at Monitorama 2016 in Portland, OR. This presentation is about using statistical information to simulate high volume data traffic during product development.
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (15)
Similar to Fake It 'Til You Make It
Similar to Fake It 'Til You Make It (20)
Recently uploaded
Recently uploaded (20)
Fake It 'Til You Make It
- 1. Fake It ’Til You Make It A case study in generating more data than your systems John Stanford -- @jxstanford
- 4. • Structure • 1 controller • 6 compute nodes • Usage • Training classes • R & D • Pipeline • 7 Heka log shippers • 1 Heka log collector • 1 Elasticsearch node shipper shipper shipper shipper shipper shipper shipper collector Elasticsearch The Environment
- 6. Every good recipe starts with onions
- 7. • 21 – 31K per hour • An obvious pattern • A dead spot • A couple spikes 7 Days
- 8. By Server • 96% from one node • 1% per remaining node • A couple not reporting
- 9. By Component • 60% • 25% • 15% • 14% • 1%
- 10. 7 Day Message Rate Recurring outlier!
- 12. Message Size
- 14. • Roughly 25K messages/hour • Controllers are 100x noisier than compute nodes • Swift generates 60% of traffic • 99.9% of the time, there were less than 65 messages/sec • There are some traffic spikes of ~1600 messages/sec • 95% of messages were less than 240 bytes • Largest message was 782 bytes
- 16. Model parameters • Flow rate • Hosts • Components • Size • Content • ?
- 17. The Scientist and Engineer's Guide to Digital Signal Processing - Steven W. Smith, Ph.D.
- 19. When you don’t put your toque on…
- 22. [flood_1ms] ip_address = “127.0.0.1:9997” sender = “tcp” pprof_file = “” encoder = “protobuf” num_messages = 0 message_interval = “1ms” max_message_size = 2800 variable_message_size = true late_bind_timestamp = true • Add flood process • Monitor everything • Repeat until it breaks The plan
- 23. heka-flood Seems like a good idea • The good: • Control over message rate • The not so good: • Not enough control over message content • Timestamps assigned at initialization (pull request forthcoming) • Couldn’t get variable message sizing to work • Here comes the meatloaf…
- 24. The result
- 25. Tactical • System sustained 4000 x ~1k msgs/sec • Collector and/or ES started to pause above that • No messages were dropped Strategic • Load tool was underqualified • Monitoring tool resolution is important for interpretation • I should prepare for presentations further in advance What did we learn?
- 26. Intuitions • 4MB/sec is: • 17K messages of 240 bytes • 240 bytes is 95th percentile • 64 msgs/sec is: • 99.9th percentile • 265 controllers • 1060 compute • 1:100 ratio Our First Model
- 27. • Find the bottlenecks • System resources? • Elasticsearch? • Collector? • Load generator? • Improve the model • Probability distributions • Noise • Real message samples • Off host load • Regime changes • Real world feedback Next Steps
Editor's Notes
- Lately, most of my time has been spent with these tools, and generally shifting to the right.
- We recently started using Heka instead of Logstash. It’s an experiment, but we’re embracing Go in some other development contexts, and we have not Ruby skills in house. Heka is interesting, but still a young project. I have high hopes for it from a performance perspective, and have had good interactions with the community for both Q & A and code acceptance.
- When sales or consultants come to me and ask how many nodes our product can support, I’d like to provide a better answer than the shrug emoji, so we’re going to walk through a modeling exercise.
- Let’s start with the big picture.
- Since the missing servers are compute nodes, they are most likely have a very similar profile as the other compute nodes.
- Query all the logs for 7 days, and do a date histogram on the timestamp with an interval of 1 second. With our amount of data, the aggregation took about 15 seconds. Your situation might be different, so you can adjust the lookback or the interval to suit your needs. We see that 99.9% of the time, there were no more than 64 messages per second, but there are a few outliers. I didn’t include the chart, but I think there were about 10 instances of a rate over 1000 messages/sec. We’ll want to consider those in a robust model.
- But for now, let’s remove them so we can see the shape of the message rate histogram. I’ve hand drawn in the red line to highlight what I think is a reasonable shape.
- Let’s do a similar analysis of the message size. Here’ well look at the last 7 days, but do a percentiles aggregation. We can see that our max message size was 782 bytes, but that 95% of messages were less than 240 byes. This is only the text component of the message, not the other fields like tags, timestamp, etc., but those things are pretty constant in this environment, so it should be ok.
- So, what did we learn out our environment?
- Now, we’ve got a bunch of ingredients, and we want to get down to the business of making a model.
- First, we might try to select a distribution that looks most like our model parameters. That’s pretty straight forward, and you can use tools like numpy and scipy to generate these curves, then take samples from the line to get a value for a parameter.
- You might want to add some noise to your parameters to give them some “texture”. For example, you might want to vary the cadence of the message stream by making a decision at each sample time about how many messages to generate, or increase or decrease the message size.
- And then all you need to do is execute flawlessly.
- But what happens when you left your not a professional data modeler, or your knives aren’t sharp, or people are pounding the forks and knives on the table… You probably end up with something like...
- Meatloaf. Arguably not horrible, but no Thomas Keller dish either.
- Let’s revisit what what we have to work with. Shippers, collectors, and a datastore. The shippers and collectors are distributed. In our case, the datastore is on the same node as the collector. It’s unlikely that an individual shipper will generate more data than the network, or an individual collector can handle, so we’ll focus on the flooding the collector from the local host, and see how the collector / elasticsearch combo handle things. There’s a chance that we can overwork the OS, but this still seems like a good starting point to isolate bottlenecks.
- Heka has a load testing utility called flood. I tend to build a mental model of what these types of utilities should do, then by the time I get a chance to prototype a solution using them, I can only hope they they are close to my mental model. In this case, I missed the mark a little, and hit a few other snags, but overall, I was able to demonstrate what I had hoped to.
- A simplified overview of what I found was that everything ramped up pretty well until I got over 5500 messages/sec, then things started to destabilize. The shippers started pausing for a few seconds at a time, then would restart, run for a while and pause again. I included two views of this behavior to show how the sample rate can mask or highlight system behavior. Marvel (on top) is aggregating over 10 seconds, Kibana (bottom) is aggregating over 1 second. You can see the destabilization in Marvel, but it’s much more obvious in Kibana.
- …. But on to our first model...
- … So, when someone asks if we can handle traffic from 100 controllers and 15000 compute nodes, I can say “The model says yes!”, but when they ask if we can handle 250 controllers and 10000 compute nodes, I can say “The model says “Oh, no. That would be a bad idea...”
- Well, we didn’t really find the bottleneck, which means we didn’t get the chance to tune it or scale it. We also have a very simple model using fairly simple tools. We could definitely improve that. Finally, we would want to take some real world samples to support or refute our model. In other words, we should continuously improvement via understanding our environment, tuning parameters, and validating our model with feedback from the real world.
- Since I paid for the right to use Meatloaf, he would like to say Thanks, and ask if you have any questions…