SplunkLive! Charlotte Bronto Software


Published on

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Introduction – BioHi, thank you, thanks Pike, my name is Doug and I’mHere to talk about how bronto uses splunk.A little bit about myself:Manager of Systems Engineering for 5 yearsMy team is responsible for all the technology for Bronto.So what does Bronto Software do?
  • Bronto’s a marketing platform which encompasses email, SMS, and social media , we have a lot of clients including PartyCity, Armani Exchange, and EtsyWe send 10’s of millions of time dependent emails per dayWhich leads us to process 100’s of millions of tracking events per day.We use splunk to index 100’s of millions of events across our infrastructure per day.
  • Our challenge was that we were trying to deal with a massive collection of disparate logs scattered throughout our infrastructure.We had rolled some in-house tools but could quickly tell this development effort would not be sustainable long term and wouldn’t scale. We recognized and agreed that something had to be done to manage all this dataWe discussed the trade offs of build vs buy?
  • Ok, so one option was to build:We looked at building our own data indexing platform using hadoop, hbase and flume to name a few. Also, we would be leaning heavily on engineering and development resources that were currently tasked with building new client facing featuresRetasking them to create internal tools We would have also incured significant cost with building our own. Build doesn’t mean cheapTriple annual license cost plus 6 months from day to hire start to implement less than 50% of functionality of SplunkClick SlideSo we looked into buying Splunk.Splunk would be more cost and time effective for engineering and development resources.Much better Speed to valueWith other products there was either signifigant challenges or was impossible to integrate into our opensource infrastructure.Splunk was very easy to setup and even dropped right into our existing puppet deploy framework.This is not a very common event for bronto, to find a 3rd party product that just drops right in.
  • The adoption of splunk was very easy within different groups at bronto, both inside and outside of engineering.Partly because of splunk’s ease of use, but also because the need to consume the information was already so great.The startup time with Splunk was very easy. The initial learning curve for new users was very short.
  • Splunk’s versatility lends itself well to solve multiple problems at Bronto.I’m sure a lot of you are using splunk to gain easy access to system level information across your environment both from logs and secondary events. And this has proven to be a huge win for us too.But specific to bronto, we use splunk to capture and correlate data for more bronto specific events like email metrics And helping us to understand and predict email deliverability trends, both good and bd.And one of the key features of splunk that’s been a huge win for us, is it’s enabled us to get people that are not necessarily technical or within the engineering organization empowered to help themselves.What I mean by that is that before splunk, if someone wanted to know what the reason code for a group of email bounces was, someone on my team had to dig through logs, or write a script to parse the logs for this specific query. This didn’t scale well at all.But by giving people access to splunk, it’s freed up a huge amount of time for my team, that we can then devote to other needs.And we use splunks alerting to let us know when things may not be going to well for deliverability, both customer caused an ESP caused. So it really helps us help our customers.
  • Dashboards have really proven to be helpful, specifically to give a quick and constant representation of how both deliverability successes and slowdowns are happening.This is one of the most important dashboards we use on a daily basis, that really wasn’t possible before splunk. It allows us to distill the deliverability data down into something quickly and easily digestible.Which helps us spot trends earlier.
  • Deliveries don’t always go according to the customer’s plan and this this graph helps us understand, very quickly what the reason is.It could be that a certain email recipient domain has decided today was the day to make a change to their acceptance rules.And again, being able to take multiple streams of analytical reporting and then concisely represent it is a great win for us and for our customers, to help us quickly respond to problems.And see how successful we are.
  • I’ve talked a lot so far about how Bronto uses splunk, and mostly about how people out side of Systems Engineering leverage splunk.But we use splunk within Engineering a lot too. And I’ve always been a sys-admin at heart.Splunk allows us to spend time doing the things we want, and need to be doing.Splunk looks at the logs for us and lets us know when there is a problem. We tried to do this with logster and nagios, but with the sheer volume of logs, it didn’t scale.And that’s on one server, let alone hundreds that need to correlate trends.Splunk is also a key component of our security infrastructure, by monitoring logins and security events and letting us know if someone trying to brute force a login for instance.And we can’t forget the Ad-hoc searching, which would have been almost impossible with the build option with hadoop. Being able to search across the infrastructure and see the logs in real time has been a huge hit.
  • I just want to get more people using splunk, there is still more that can be gleaned, more stones can be turned over.Overall we’re really happy with splunk, but we’re always looking to improve and grow with splunk.
  • So that’s the overview of how Bronto uses splunk.Thank you for having me, and if you have any questions I’ll be around for the rest of the day.
  • SplunkLive! Charlotte Bronto Software

    1. 1. Doug HairfieldManager Systems Engineering
    2. 2. Charlotte, March 29, 2012 2 Copyright © 2012, Splunk Inc.
    3. 3. Challenge: Dealing with Massive Amount of Data • We send over 55 million emails per day • How do we parse and correlate logs from the entire stack • Build or buy “Splunk is so powerful and easy to use.” Charlotte, March 29, 2012 3 Copyright © 2012, Splunk Inc.
    4. 4. • Heavy use of or • 1 engineer to deploy engineering resources • Immediate return• Long return cycle • Actually cost less than build• Expensive $$$ • Fit in our open-source• Dependent on bleeding environment edge software • Compatible with existing (flume, hbase, hadoop) framework Charlotte, March 29, 2012 4 Copyright © 2012, Splunk Inc.
    5. 5. Who Uses Splunk?• Systems Engineering• Production Operations• Software Developers• Client Services• Email Deliverability Teams Charlotte, March 29, 2012 5 Copyright © 2012, Splunk Inc.
    6. 6. How Bronto Splunks• Tracking email metrics• Email success trends• System level info and alerting• Know when things go wrong Charlotte, March 29, 2012 6 Copyright © 2012, Splunk Inc.
    7. 7. Dashboards• Overall domain health• Bounces per: • Domain • Hour • Mail service provider• Who’s the problem child? Charlotte, March 29, 2012 7 Copyright © 2012, Splunk Inc.
    8. 8. Deliverability Reporting• Top blocks by reason code and host• Alerting for block thresholds• Reduced response time to mitigate customer impact Charlotte, March 29, 2012 8 Copyright © 2012, Splunk Inc.
    9. 9. Systems and Ops• System log alerting• Constant security auditing• Ad-hoc searching for event correlation. (Ops loves this one) Charlotte, March 29, 2012 9 Copyright © 2012, Splunk Inc.
    10. 10. Whats Next?• Educate and evangelize.• Apply Splunk to our application and system security layer.• Build more alerting and dashboards for system level logs. “Theres nothing-else out there like Splunk – Splunk’s Badass!” Charlotte, March 29, 2012 10 Copyright © 2012, Splunk Inc.
    11. 11. Thank you!