Denver Big Data Analytics Day
•
0 likes•450 views
Many companies have discovered that there is “gold” in their server log files and machine data. Closely monitoring this data can improve security, help prevent costly outages and reduce the time it takes to recover from a problem. In this presentation, GTRI’s Micah Montgomery explains how operational intelligence can be gained from machine data, and how Splunk Enterprise can turn this data into actionable insights. Also presenting was NetApp’s Steve Fritzinger, who discussed how to manage the challenges of capturing and storing a flood of data without breaking the bank. Presented at "Denver Big Data Analytics Day" on May 18, 2016 at GTRI.
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (11)
Similar to Denver Big Data Analytics Day
Similar to Denver Big Data Analytics Day (20)
More from Zivaro Inc
More from Zivaro Inc (20)
Recently uploaded
Recently uploaded (20)
Denver Big Data Analytics Day
- 1. WWW.GTRI.COM Denver Big Data Analytics Day See the Forest and the Trees © 2016 Global Technology Resources, Inc. All rights reserved.
- 2. Agenda
- 3. GTRI Splunk Practice Overview Highlights: • Splunk’s 1st Elite Partner and one of only two Splunk Certified Training Centers in the U.S. • GTRI provides end-to-end support for Splunk from pre-sales engineering to post-sales professional services, implementation, training and optimization • Splunk’s most credentialed partner in North America: o GTRI holds over 90 Splunk Certifications: 8 Certified Architects 16 Certified Solutions Engineers (SE-I & SE-2) Certified Training Center
- 4. GTRI Solution Areas & Capabilities GTRI Solution Areas
- 5. So What is Big Data Anyway?
- 7. Turn Machine Data into Operational Intelligence INDEX ANY MACHINE DATA: ANY SOURCE, TYPE, VOLUME Online Services Web Services Servers Security GPS Location Storage Desktops Networks Packaged Applications Custom ApplicationsMessaging Telecoms Online Shopping Cart Web Clickstreams Databases Energy Meters Call Detail Records Smartphones and Devices RFID On- Premises Private Cloud Public Cloud GAIN REAL-TIME VISIBILITY Application Delivery Security and Compliance Infrastructure Monitoring Business Analytics Internet of Things 7
- 8. Splunk Company Overview 8 Company • Global HQs: San Francisco London Hong Kong • 2,100+ employees globally • Annual Revenue: $668.4M (YoY +48%) • NASDAQ: SPLK Products • Free trial to massive scale • Splunk products: Splunk Enterprise Splunk Cloud Hunk Splunk Light Splunk MINT Premium Solutions Customers • 11,000+ customers • Across 110+ countries • Small to large organizations • More than 80 of the Fortune 100 • Largest license: 1+ Petabyte/day
- 9. 2015 U.S. Award Wins “Splunk is honored to once again be recognized as one of the Bay Area’s Best Places to Work. We know our team is paramount to our continued success so we are deeply invested in individual development, career growth and a workplace environment that fosters our fun, passionate and collaborative community of Splunkers.” – Godfrey Sullivan, chairman and CEO, Splunk, San Francisco Business Times Best Places to Work 2015
- 10. 11 Splunk Enterprise Breakthrough Performance & Scale Doubles performance and lowers TCO • 2x Search & Indexing Speed • 20-50% Increased Capacity • 20%+ Reduced TCO Meeting the needs of the most demanding organizations Advanced Analysis & Visualization High-Volume Event Collection Enterprise-Scale Platform Supports DevOps and IoT data analysis at scale Simplifies analysis of large datasets Delivers Enterprise platform requirements • Anomaly Detection • Geospatial Mapping • Single-Value Display • HTTP Event Collector • Developer API & SDKs • 3rd Party Integrations • Expanded Management • Custom Alert Actions • Data Integrity Control
- 11. Demo © 2016 Global Technology Resources, Inc. All Rights Reserved.12
- 12. WWW.GTRI.COM NetApp Solutions for Splunk Steve Fritzinger, Americas Channel Emerging Products BDM © 2016 Global Technology Resources, Inc. All rights reserved.
- 13. Splunk Data Requirements • Very high index rates o Hundreds of GB to tens of TB per day • Age/usage based data tiering o Hot, warm, cold and frozen buckets • Complex data compliance and privacy requirements • Application manages data availability o Compression, back-up, archiving, DR • Long retention requirements o Months to years o Potentially PB of storage required 14 © 2016 NetApp, Inc. All rights reserved. NetApp Confidential – Limited Use
- 14. 15 © 2016 NetApp, Inc. All rights reserved. NetApp Confidential – Limited Use
- 15. NetApp E-Series big data product portfolio Midrange (E5000 Series) NetApp E5600 3PB 12GBps (reads) 6GBps (writes) 825,000 IOPS (max. sustained) NetApp® E2700 1.5PB 8GBps (reads) 2.1GBps (writes) 80,000 IOPS (max. sustained) Entry (E2000 Series) All Flash (EF500 Series) NetApp EF560 384TB 300u sec latency 12GBps (bandwidth) 825,000 IOPS (max. sustained) 6 © 2016 NetApp, Inc. All rights reserved. NetApp Confidential – Limited Use
- 16. NetApp Solution for Splunk Splunk Cluster Environment with 8 Indexers, 3 Forwarders, 1 Search Head Function1 NetApp Splunk testing November 2015 Internal Drives NetApp E-Series Hot and Warm Buckets 32 SSD 24 SSD Useable Flash Capacity 12TB 12TB Cold Bucket 80 SAS 24 SAS RAID Configuration RAID 10 Dynamic Disk Pools Replicas 3 2 E-Series Configuration 10 © 2016 NetApp, Inc. All rights reserved. NetApp Confidential – Limited Use
- 17. NetApp E-Series E5600 vs Internal DAS 18 69%Average Search Performance Gain 2 Copies of Data © 2016 NetApp, Inc. All rights reserved. NetApp Confidential – Limited Use
- 18. Benchmark Results—Baseline E-Series vs Commodity Baseline Dense Baseline very Dense Baseline Rare Baseline Very Rare Stream Search (sec) E-Series Commodity Server w/ Internal Disk 12 © 2016 NetApp, Inc. All rights reserved. NetApp Confidential – Limited Use
- 19. Improving Analytics with lower TCO Cellular Service Provider using Splunk • Operational platform capable real time and batch oriented analytics • Performance Improvements while reducing server count 5:1 • Expandable to Hot/Warm/Cold Data buckets if warranted 100 Indexers 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server Splunk indexing requires performance • EF560 provides ultra low latency performance • Exceeded the performance of 100 indexers with DAS • Fault tolerant (2-copy) indexer architecture • Indexing more with less hardware and better reliability E-Series 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server 1U Server E-Series 20 Indexers Before After 13 © 2016 NetApp, Inc. All rights reserved. NetApp Confidential – Limited Use
- 20. Baseline Results—Baseline E-Series vs Commodity 21 Baseline Dense Baseline Very Dense Baseline Rare Baseline Very Rare Static Search (sec) during Splunk Cluster Node Failure E-Series Commodity Server w/ Internal Disk 64% Average Gain in failure case © 2016 NetApp, Inc. All rights reserved. NetApp Confidential – Limited Use
- 21. NetApp E-Series • Scale system performance and storage independently • Easier long term maintenance and life cycle management • Overall TCO better than White Box at scale • Encrypt data at rest • Data Management geared for Analytics o One architecture for Hot, Warm, Cold and Frozen data tiers o Flash, Hybrid for Hot E5600 for Warm E2760 for Cold o Better reliability and performance under failure Performance, efficiency, reliability FY 2015, Q2 Earnings Announcement 22 For customers who are increasing their cyber security defenses with real-time analysis, the E-Series capability to store hot and cold data under the same data management architecture substantially improves the efficiency and flexibility of their environments. George Kurian, CEO NetApp © 2016 NetApp, Inc. All rights reserved. NetApp Confidential – Limited Use
- 22. WWW.GTRI.COM Use Cases A Petabyte of data is a terrible thing to waste © 2016 NetApp, Inc. All rights reserved. --- NETAPP CONFIDENTIAL --- 2 3
- 23. Use Cases • Transportation Industry o New York Airbrakes • Public Sector o Premier Space Launch Partner • Technology Sector o Cisco • Online Retail o TicketMaster © 2016 NetApp, Inc. All rights reserved. --- NETAPP CONFIDENTIAL ---24
- 24. New York Airbrake © 2016 NetApp, Inc. All rights reserved. --- NETAPP CONFIDENTIAL --- 25 http://www.splunk.com/en_us/customers/success-stories/new-york-air-brake.html
- 25. Use Case – Premier Space Launch Provider • Security tool to manage increasingly complex environment and monitor for APTs • Centralized secure logging solution to meet compliance requirements. • 65 + Unique Data Sources to Aggregate and Monitor • Multiple major data centers as well as remote offices. • Over 2,000 server systems and 5,000 end user workstations.
- 27. Splunk Solution – Premier Space Launch Provider • Multi-phase Splunk installation including Enterprise Security App. • Phase 1 - High priority data types and the Splunk infrastructure. • Phase 2 - Correlation across data types, monitoring solutions and locations. • 200GB initial daily volume license. • Redundant architecture centralizing data into the primary and secondary data centers.
- 28. Replacing a SIEM @ Cisco Challenges • Security Information and Event Management System could not meet security needs • Very difficult to index non-security or custom app log data • Serious scale and speed issues. 10GB/day and searches took > 6 minutes • Difficult to customize, reliance on pre-built rules which generated false positives Splunk Solution • Easy to index any type of machine data from any source • Over 60 simultaneous users, correlations, reporting, advanced threat detection • Use all data + flexible searches and reporting = empowered team • 900 GB/day and searches take < minute. 7 global data centers with 350TB store • Estimated that Splunk is 25% the cost of a traditional Security Information and Event Management System “We moved to Splunk from traditional SIEM as Splunk is designed and engineered for “big data” use cases. Our previous SIEM was not and simply could not scale to the data volumes we have. “ - Gavin Reid, Leader, Cisco Computer Security Incident Response Team
- 29. Ticketmaster Business challenge Ticketmaster needed a faster way to detect and block ticket scalpers who used bots to purchase volumes of tickets and drove up prices for event- goers An unplanned new business value was discovered whereby promoters could see venue ticket sales in real time and decide whether to add more concert dates NetApp metrics Nine NetApp® E5460 systems SAS-attached in an enterprise building-block architecture Solution stack Splunk NOC dashboards for capacity problems, availability issues, forensics, transaction tracing, counts, durations, and failed transactions Why NetApp E-Series Extreme performance Cost-effective, simple deployment and high availability © 2016 NetApp, Inc. All rights reserved. NetApp Confidential—Limited Use30 Ticketmaster is a Live Nation Entertainment company that sells $1 million worth of tickets per minute. That’s more tan $16 billion annually out of 7 data centers worldwide
- 30. WWW.GTRI.COM Thank You © 2016 Global Technology Resources, Inc. All rights reserved.
Editor's Notes
- What is not big data - data warehouses/database. They cannot take in all the original data. Also often batch-oriented, not real-time. Explain the key security use cases
- Our customers typically start with Splunk to solve a specific problem, and then expand from there to address a broad range of use cases, across application troubleshooting, IT infrastructure monitoring, security, business analytics, Internet of things, and many others that are entirely innovated by our customers. Here’s how it works. Splunk software and cloud services reliably collect and index machine data, from a single source to tens of thousands of sources. All in real time. - Once data is in Splunk, you can search, analyze, report-on and derive insights from all your data - across real-time or historical data that may be stored in Hadoop or other NoSQL data sources.
- Splunk has more than 2,100 employees worldwide, with our global headquarters in San Francisco. Our 11,000+ customers in over 110 countries are using Splunk software and cloud services to improve service levels, reduce operations costs, mitigate security risks, enable compliance, enhance DevOps collaboration and create new product and service offerings. Our products are designed to fit your needs and are built to be as frictionless to deploy as possible. Simple download Splunk software, or sign up for the online sandbox, point it at your data, and you’ll up and running in minutes. Please always refer to latest company data found here: http://www.splunk.com/company.
- U.S. Awards won by Splunk in 2015: IoT Evolution Magazine Awards SD Times 100: Top of the Heap The CRN Women of the Channel Award CRN Big Data 100 Winner SC Magazine Awards 2015 Bay Area’s Best Place to Work CRN 5-Star Partner Program Guide Winner Healthcare Informatics Leading Edge award Cybersecurity 500 2015 CRN Channel Chiefs Fast Company’s Ten Most Innovative Companies in Big Data CRN 100 Coolest Cloud Computing Vendors of 2015 Trend-setting Products in Data and Information Management for 2015 CSO50 Awards
- More than 10,000 customers in 100 countries have purchased the enterprise license of Splunk. This includes a majority of the Fortune 100. Enterprises, service providers and government agencies in 100 countries use Splunk to improve service levels, reduce IT operations costs, mitigate security risks and drive new levels of operational visibility. As they gain new visibility into their real-time and historical machine data, Splunk’s customers are finding answers and solving the most challenging issues facing IT and the business.
- Splunk is the industry-leading platform for Operational Intelligence, delivering both cloud and on-premise solutions tailored to meet the needs of any size organization. Splunk is increasingly being used as a mission-critical, enterprise-wide operational intelligence source, processing 100's of terabytes of data per day. Release 6.3 continues our journey to support the ever-expanding requirements of the most demanding organizations Release 6.3 is especially targeted to meet their needs for scalability and management, extended analysis features, analysis of high-volume data from application and IoT events, and new flexible connectivity options to their business and operational systems. Release 6.3 is a platform release. All 6.3 features are supported on Splunk Enterprise, most on Splunk Cloud, and select features are supported on the Hunk and Splunk Light products
- Cisco is another great story. Their traditional SIEM had issues getting in any kind of data source. It was hard to customize. It had scale and speed issues, which we will detail on the next slide. They literally looked at 9 other SIEMs/logging tools and Splunk was the clear winner. So they use us now for incident investigations, correlations, reporting. Worldwide, they are indexing over 1 terabyte (TB) a day. They have over 150 TBs stored in Splunk, 25 indexers, in 7 locations across the world. And they estimate it is 25% of the SIEM cost. So great value for Cisco.