Your SlideShare is downloading. ×
Conf2013 bchristensen thebig_t
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Conf2013 bchristensen thebig_t

198
views

Published on

Splunk Conf 2013 September 30-October 3 & Splunklive Denver. …

Splunk Conf 2013 September 30-October 3 & Splunklive Denver.

Monitoring for the big "T". Learn how Ping Identity manages, deploys and monitors it's hybrid cloud SaaS applications using best of breed solutions. Tools and people create T = r + t, our philosophy for transparency and reliability.

Published in: Technology

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
198
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
5
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Copyright ©2013 Ping Identity Corporation. All rights reserved.
  • 2. Infrastructure Operations Copyright ©2013 Ping Identity Corporation. All rights reserved.
  • 3. Monitoring for the big T. Copyright ©2013 Ping Identity Corporation. All rights reserved.
  • 4. About Me Copyright ©2013 Ping Identity Corporation. All rights reserved.
  • 5. 3rd Generation Computer Geek Roy Christensen: 35 years 3C > HoneyWell > Digital Equipment Corp Mike Christensen: 40 years DEC > Compaq> Hewlett Packard Beau Christensen: 17 years AstraZeneca > Send.com > DirecTV > Ping Identity 92 years family experience (hah) Copyright ©2013 Ping Identity Corporation. All rights reserved.
  • 6. About Ping Identity • We believe secure professional and personal identities underlie human progress in a connected world. Our purpose is to enable and protect identity, defend privacy and secure the Internet. • Over 1,000 companies, including over half of the Fortune 100, rely on our awardwinning products to make the digital world a better experience for hundreds of millions of people. • Denver, Colorado. Est. 2003 Copyright ©2013 Ping Identity Corporation. All rights reserved.
  • 7. Infrastructure Operations @Ping Site Reliability Engineering (SRE) Production Web Operations Ops to On-Demand Dev Configuration Engineering (CFE) Automation, Deployment, Labs Ops to On-Premise Dev Infrastructure Engineering (IFE) Iron, Network, Security Ops to Support & Helpdesk Copyright ©2013 Ping Identity Corporation. All rights reserved.
  • 8. (Quickie) Architecture • • • • • Hybrid Cloud Application VMware/AWS SOA (lots of little services) Red/Black Deployment Cassandra & Galera MySQL Copyright ©2013 Ping Identity Corporation. All rights reserved.
  • 9. Splunk Systems • 7 Indexers • Distro search across 3 regional data centers • ~ 40Gib/day • No clustering (yet) • Universal FWD installed in all templates • Moving all Splunk to clustered architecture in VPCs. Copyright ©2013 Ping Identity Corporation. All rights reserved.
  • 10. Trust (the big T) Copyright ©2013 Ping Identity Corporation. All rights reserved.
  • 11. Because uptime is dead. http://www.kitchensoap.com/2013/01/03/availability-nuance-as-a-service/ Copyright ©2013 Ping Identity Corporation. All rights reserved.
  • 12. http://uptime.pingidentity.com Copyright ©2013 Ping Identity Corporation. All rights reserved.
  • 13. http://uptime.pingidentity.com Copyright ©2013 Ping Identity Corporation. All rights reserved.
  • 14. What Builds Trust? Copyright ©2013 Ping Identity Corporation. All rights reserved.
  • 15. Lots of Things… Security Honesty Reliability Integrity Dependability Maturity Transparency Copyright ©2013 Ping Identity Corporation. All rights reserved.
  • 16. We rolled those up into two. Reliability & Transparency Copyright ©2013 Ping Identity Corporation. All rights reserved.
  • 17. Reliability: Keep Shit Running Copyright ©2013 Ping Identity Corporation. All rights reserved.
  • 18. Keeping Shit Running • Architecture – – – • Tools – – – • SOA (duh!!!!) Highly Automated Deployments Active/Active, quick failover, multi-region We use a ton of tools We constantly question the tools we use We are always looking for new tools Security – – – Eyes always on Scanners running continuously Constant Remediation Copyright ©2013 Ping Identity Corporation. All rights reserved.
  • 19. Transparency: Talk About Shit Copyright ©2013 Ping Identity Corporation. All rights reserved.
  • 20. Talking About Shit • Talk Publicly – – – – • When Stuff Breaks, Talk More – – – • Talk about decisions we make Talk about architecture Talk to vendors Talk to customers Quick & dirty status updates Verbose post mortems Be honest Metrics – – – Use 3rd Parties Expose as much monitoring as you can Make it relevant to customers Copyright ©2013 Ping Identity Corporation. All rights reserved.
  • 21. T=r+t Trust = reliability + transparency Copyright ©2013 Ping Identity Corporation. All rights reserved.
  • 22. Trust is a Zero Sum Game Time Trust Copyright ©2013 Ping Identity Corporation. All rights reserved.
  • 23. T = +2 Time Trust Copyright ©2013 Ping Identity Corporation. All rights reserved.
  • 24. Copyright ©2013 Ping Identity Corporation. All rights reserved.
  • 25. Time Trust T = -2 Copyright ©2013 Ping Identity Corporation. All rights reserved.
  • 26. Time Trust T=0 Copyright ©2013 Ping Identity Corporation. All rights reserved.
  • 27. Monitoring is the foundation of Trust. Copyright ©2013 Ping Identity Corporation. All rights reserved.
  • 28. is the backbone of our monitoring strategy. Copyright ©2013 Ping Identity Corporation. All rights reserved.
  • 29. <3 Monitoring The right tool for the right job = Lots of tools. Copyright ©2013 Ping Identity Corporation. All rights reserved.
  • 30. Copyright ©2013 Ping Identity Corporation. All rights reserved.
  • 31. Lots of tools = Best of Breed, or Stack Monitoring Copyright ©2013 Ping Identity Corporation. All rights reserved.
  • 32. Let’s build the stack. Copyright ©2013 Ping Identity Corporation. All rights reserved.
  • 33. Transport & Network Layer. Fundamental interactions between services. Extremely important to understand, forms the basis of troubleshooting efforts. “Can you ping it? Tracert? Tcpdump?” Copyright ©2013 Ping Identity Corporation. All rights reserved.
  • 34. Instance Resource Monitoring Historical resource consumption, collected and correlated with incidents and system events. Scalability intelligence. “Traditional” systems monitoring. Nagios, Zenoss, Zabbix, etc. Copyright ©2013 Ping Identity Corporation. All rights reserved.
  • 35. Machine Data, Logs. Extraordinary versatility: events, alerting, reporting, security, business metrics, and OI. Allows the creation of dashboards and event types relative to your own environment. Events, logs, traditional “syslog stuff.” Copyright ©2013 Ping Identity Corporation. All rights reserved.
  • 36. Application Performance Monitoring Detailed view of how the applications are running on top of the rest of the stack. Identify bottlenecks, architecture issues, and accurately model user experience with RUM. “Page loads are slow.” “Why are there SQL queries that return 30,000 rows?” Copyright ©2013 Ping Identity Corporation. All rights reserved.
  • 37. Availability Monitoring External agents monitor services from multiple locations around the globe. Use application heartbeats, not ICMP or TCP. Provides near real-time alerting and is immediately visible to the customer. “The site is down.” Copyright ©2013 Ping Identity Corporation. All rights reserved.
  • 38. Splunk is the system’s mortar. Binding systems together, filling gaps, creating stability. Copyright ©2013 Ping Identity Corporation. All rights reserved.
  • 39. Security Quick visual impact. Easy to identify location, frequency, and network address of aggressor. Quick drill down into detailed Splunk searches. Copyright ©2013 Ping Identity Corporation. All rights reserved.
  • 40. Traffic Real-time traffic maps are immediately recognizable to everyone in the organization. Copyright ©2013 Ping Identity Corporation. All rights reserved.
  • 41. Operations Operational dashboards provide a quick overview of system health by displaying error correlation with traffic levels and event type heat maps. Copyright ©2013 Ping Identity Corporation. All rights reserved.
  • 42. Global Load Balance Shows traffic distribution between production data centers. Copyright ©2013 Ping Identity Corporation. All rights reserved.
  • 43. Maximize Machine Data Layer Versatility Build your own dashboards (it’s easy). Maintain proper naming conventions: na-den-www-13-8-17-0ec5e8da-128-193 Design for wetware. Ask for feedback. Maintain event types. Copyright ©2013 Ping Identity Corporation. All rights reserved.
  • 44. Having tons of tools is great. Each layer of the stack is matched well with the system’s requirements… Aggregation of metrics is the new challenge. How do you see it all? Copyright ©2013 Ping Identity Corporation. All rights reserved.
  • 45. Stack Integration Copyright ©2013 Ping Identity Corporation. All rights reserved.
  • 46. Stack Integration = MTTR Speed eventtype=critical.* Application integration allows speedier access to event timeline and context of the alarm using markers. Allows SRE to gather relevant information quicker, using the same tools with less screens. Copyright ©2013 Ping Identity Corporation. All rights reserved.
  • 47. Splunk has a TON of integrations. Copyright ©2013 Ping Identity Corporation. All rights reserved.
  • 48. See? --> Copyright ©2013 Ping Identity Corporation. All rights reserved.
  • 49. Always Need More. Copyright ©2013 Ping Identity Corporation. All rights reserved.
  • 50. Remember… T=r+t Copyright ©2013 Ping Identity Corporation. All rights reserved.
  • 51. Keep Shit Running. Copyright ©2013 Ping Identity Corporation. All rights reserved.
  • 52. Talk About Shit. Copyright ©2013 Ping Identity Corporation. All rights reserved.
  • 53. Copyright ©2013 Ping Identity Corporation. All rights reserved.
  • 54. Use Splunk. Copyright ©2013 Ping Identity Corporation. All rights reserved.
  • 55. Questions? Copyright ©2013 Ping Identity Corporation. All rights reserved.
  • 56. Thanks! bchristensen@pingidentity.com @beauchristensen www.pingidentity.com/blogs http://status.pingidentity.com Copyright ©2013 Ping Identity Corporation. All rights reserved.

×