27. S3MPER
S3mper is a library that provides an additional layer of consistency checking on top of Amazon's S3 index through use of a consistent, secondary index.
S3mper is a library that provides an
additional layer of consistency
checking on top of Amazon's S3 index through
use of a consistent, secondary index.
47. • Ad-hoc query with different dimensions
• Quick aggregations and Top-N queries
• Time series with flexible filters
• Quick access to raw data using boolean
queries
What We Need
56. RequestId Parent Id Node Id Service Name Status
4965-4a74 0 123 Edge Service 200
4965-4a74 123 456 Gateway 200
4965-4a74 456 789 Service A 200
4965-4a74e 456 abc Service B 200
Status:200
Something happened. Our traffic turned into a hockey stick, and the number of applications exploded. So, log traffic also exploded. Simple log scraping wouldn’t cut it any more.
For one thing: interactive exploration. Sometimes we want to get data in real time so we can act quickly. Some data is only useful in a small time window after all. Sometimes we want to perform lots of experimental queries just to find the right insights. If we wait too long for a query back, we won’t be able to iterate fast enough. Either way, we need to get query results back in seconds.
Here is one example: we process more than 150 thousand events per second about user activities. What if we’d like to know the geographically how many users started playing videos in the past 5 minutes? So I submit my query, and in a few seconds....
But this is an aggregated view. What if I want to drill down the data immediately along different dimensions? In this particular case, to find out failed attempts on our SilverLight players that run on PCs and Macs?
Note this is different from alerting based on monitoring metrics. Monitoring metrics are great and versatile. But it doesn’t help us catch unexpected errors. When we build an application, we instrument our code diligently, yet it’s very likely we miss some critical instrumentation points. There’s one thing that we always catch, though: logged errors and unhandled exceptions. It’s about The alert provides a precise entrypoint and the right context for people to drill down the right problems
Note this is different from alerting based on monitoring metrics. Monitoring metrics are great and versatile. But it doesn’t help us catch unexpected errors. When we build an application, we instrument our code diligently, yet it’s very likely we miss some critical instrumentation points. There’s one thing that we always catch, though: logged errors and unhandled exceptions. It’s about The alert provides a precise entrypoint and the right context for people to drill down the right problems
