Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

ABD208_Cox Automotive Empowered to Scale with Splunk Cloud & AWS and Explores New Innovation with Amazon Kinesis Firehose

1,029 views

Published on

"In this session, learn how Cox Automotive is using Splunk Cloud for real time visibility into its AWS and hybrid environments to achieve near instantaneous MTTI, reduce auction incidents by 90%, and proactively predict outages. We also introduce a highly anticipated capability that allows you to ingest, transform, and analyze data in real time using Splunk and Amazon Kinesis Firehose to gain valuable insights from your cloud resources. It’s now quicker and easier than ever to gain access to analytics-driven infrastructure monitoring using Splunk Enterprise & Splunk Cloud.

Session sponsored by Splunk"

  • Be the first to comment

  • Be the first to like this

ABD208_Cox Automotive Empowered to Scale with Splunk Cloud & AWS and Explores New Innovation with Amazon Kinesis Firehose

  1. 1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Cox Automoti ve E mpowered to Scal e wi th Spl unk Cl oud & AWS and Expl ores New Innovati on wi th Amazon Ki nesi s F i rehose R a y Z h u | P r o d u c t M a n a g e r , A W S E l i a s H a d d a d | P r o d u c t M a n a g e r , S p l u n k S t e v e n H a t c h | E n t e r p r i s e L o g g i n g S e r v i c e s M a n a g e r N o v e m b e r 2 8 , 2 0 1 7
  2. 2. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Agenda • What is Splunk • Current Splunk ingestion landscape for AWS • Current challenges • New solution • Cox Automotive use case • Q&A
  3. 3. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Forward-looking statements During the course of this presentation, we may make forward-looking statements regarding future events or the expected performance of the company. We caution you that such statements reflect our current expectations and estimates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward-looking statements, please review our filings with the SEC. The forward-looking statements made in this presentation are being made as of the time and date of its live presentation. If reviewed after its live presentation, this presentation may not contain current or accurate information. We do not assume any obligation to update any forward-looking statements we may make. In addition, any information about our roadmap outlines our general product direction and is subject to change at any time without notice. It is for informational purposes only and shall not be incorporated into any contract or other commitment. Splunk undertakes no obligation either to develop the features or functionality described or to include any such feature or functionality in a future release. Splunk, Splunk>, Listen to Your Data, The Engine for Machine Data, Splunk Cloud, Splunk Light and SPL are trademarks and regi stered trademarks of Splunk Inc. in the United States and other countries. All other brand names, product names, or trademarks belon g to their respective owners. © 2017 Splunk Inc. All rights reserved.
  4. 4. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Turn data into answers with Splunk
  5. 5. © 2017 SPLUNK INC. Machine data Your data is talking. Are you listening?
  6. 6. © 2017 SPLUNK INC. What is machine data? Application data Mobile app and website data 44% of the world’s population will own smartphones in 2017* IT infrastructure data Network servers, cloud services Average cost of downtime for a data center is $7,900 per minute* Security data Firewall data, endpoint data Cybercrime will cost the world in excess of $6 trillion annually by 2021* Customer-generated data Social media data, support call logs There are 2.8 billion social media users worldwide* Internet of Things data Temperature control, speed instruments 8.4 Billion connected "things" will be in use in 2017*
  7. 7. © 2017 SPLUNK INC. To meet higher customer expectations, companies must leverage their machine data. I expect recommendations & upgrades based on my preferences I expect seamless, personalized interactions I expect you to know who I am, across channels I expect you to proactively resolve my issues I expect you to know where I am & what I need Use machine data. Meet customer expectations. I expect you to protect my data
  8. 8. © 2017 SPLUNK INC. Why is this so hard? Machine data is messy and unpredictable Requires massive scale You don’t always know which questions to ask
  9. 9. © 2017 SPLUNK INC. Any question, any data, in real time Single Platform, Many Lenses Open Ecosystem HybridPerformance at Scale Machine Learning
  10. 10. © 2017 SPLUNK INC. © 2017 SPLUNK INC.
  11. 11. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Proven at 13,000+ customers in 110+ countries More than 85 of the Fortune 100 Technology Travel and Leisure ManufacturingGovernment Education Healthcare Telecommunications Energy and Utilities Media Financial Services and Insurance Retail Cloud and Online Services
  12. 12. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Splunk portfolio of AWS solutions AMI on AWS Marketplace Benefits of Splunk Enterprise as SaaS AMI on AWS Marketplace App for AWS AWS Integrations AWS Lambda, AWS IoT, Amazon Kinesis, Amazon EMR, Amazon EC2 Container Service SaaS Contract Billed through Marketplace Available on Splunk Enterprise, Splunk Cloud and Splunk Insights for Cloud Monitoring End-to-end AWS visibility Self-deployed AMIs or SaaS on AWS Marketplace AWS-based SaaS sdInsights for AWS Cloud Monitoring
  13. 13. © 2017 SPLUNK INC. 13 Splunk Cloud available worldwide Monitored by
  14. 14. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Current Splunk end-to-end landscape for AWS v1.2
  15. 15. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Poll-based ingestion challenges • Reliability, scalability, and fault tolerance • Management overhead of data-collection nodes • Delayed event delivery due to poll-based ingestion • API throttling with poll-based data ingestion
  16. 16. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Need for new solution
  17. 17. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Kinesis Amazon Kinesis Firehose Easily load streaming data into AWS Amazon Kinesis Analytics Easily process and analyze streaming data with standard SQL Amazon Kinesis Streams Build custom applications that process and analyze streaming data
  18. 18. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Current state of Amazon Kinesis Firehose Ingest Transfor m Deliver Amazon Kinesis Agent Amazon Kinesis Streams Amazon CloudWatch Logs Amazon CloudWatch Events AWS IoT Amazon S3 Amazon Redshift Amazon Elasticsearch
  19. 19. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Our answers to challenges • Reliability, scalability, and fault-tolerance challenges  Extremely reliable with underlying infrastructure operating in three different AZs  Extremely durable with three copies of same data in three different AZs  Temporarily holds and buffers data to absorb back pressure  Data backup to Amazon S3 upon failure • Management overhead of data-collection nodes in existing solution  Serverless with no resource provision or management overhead • Delayed event delivery due to poll-based ingestion  Push delivery with configurable buffer size and interval • API throttling with poll-based data ingestion  Horizontally scalable with no limit
  20. 20. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Kinesis Firehose with Splunk delivery AWS IoT Amazon S3 Amazon Redshift Amazon Elasticsearch Amazon Kinesis Agent Amazon Kinesis Streams Amazon CloudWatch Logs Amazon CloudWatch Events Ingest Transform Deliver
  21. 21. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Kinesis Firehose advantages Why would I use Amazon Kinesis Firehose as opposed to other ingestion mechanisms for Splunk?
  22. 22. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Why Amazon Kinesis Firehose • Fully managed service with serverless architecture • Bypass the need for setting up and managing heavyweight forwarder • Greater reliability and scalability • Well integrated with various data sources • Easy to use with no programming requirement • Ability to transform raw data prior to sending it to Splunk • Very low cost—$0.029 per GB of data ingested via Amazon Kinesis Firehose
  23. 23. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Serverless and scalable • Supports native balancing to indexing tier • Supports Splunk Cloud and Splunk Enterprise • Leverages HTTP Event Collector’s indexer acknowledgment • Makes sure events are written to disk on Splunk’s side • Unacknowledged events are sent again
  24. 24. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Serverless and scalable • Supports ELB and third-party load balancers
  25. 25. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Greater reliability: Hybrid push/pull • Supports delivery acknowledgment. Unacknowledged events can be persisted to Amazon S3 and ingested again via alternative delivery mechanism. • Undelivered and unacknowledged events can be ingested from Amazon S3 bucket using poll-based mechanism (Splunk add-on for AWS) • Hybrid push/pull: backup route less prone to same bottleneck
  26. 26. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Greater reliability: Serverless architecture • Undelivered and unacknowledged events can be ingested from Amazon S3 using AWS Lambda for full push-based architecture • Lambda can be configured to push data to a failover HEC endpoint • Pure push, serverless-based architecture; no forwarder required
  27. 27. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. • Consolidate Amazon VPC flow data from multiple accounts into one Amazon Kinesis Firehose delivery stream • Ability to route events to different indexes based on AWS Lambda conditions Cross-account delivery
  28. 28. New Splunk end-to-end landscape for AWS
  29. 29. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Kinesis Firehose use case When would I use Amazon Kinesis Firehose as opposed to other ingestion mechanisms for Splunk?
  30. 30. Steven Hatch 30 Leading international Splunk rollout across Cox Automotive 20 Years Enterprise IT Experience Enterprise Logging Services Manager
  31. 31. A leading provider of products and services that span the automotive ecosystem worldwide. More than 20 brands that together provide end-to-end solutions for customers large and small. A subsidiary of Cox Enterprises, Inc.
  32. 32. Transform the way the world buys, sells, and owns cars.
  33. 33. Audio / Video Network Gear Monitoring
  34. 34. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Internet Data Center App Closed Internet Auction Lane
  35. 35. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Internet Data Center App Closed Internet
  36. 36. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
  37. 37. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Monitoring App Monitoring App Monitoring AppMonitoring App
  38. 38. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Correlation Alert routing Ingest all component data Homegrown anomaly detection
  39. 39. Wrapping intelligence around noise
  40. 40. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Expansion of self-service @ Cox Automotive Build recipe of success by way of the key ingredients Devs or application teams can build it however they want
  41. 41. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Center of Excellence Build partnerships Develop standards Standardize infrastructure Empower, educate, and delegate Evangelize Build for scale Grow the business
  42. 42. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Q&A
  43. 43. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Thank you!

×