Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Log Analytics Optimization

3,315 views

Published on

Learn more: http://hortonworks.com/hdf/
Log data can be complex to capture, typically collected in limited amounts and difficult to operationalize at scale. HDF expands the capabilities of log analytics integration options for easy and secure edge analytics of log files in the following ways:

More efficient collection and movement of log data by prioritizing, enriching and/or transforming data at the edge to dynamically separate critical data. The relevant data is then delivered into log analytics systems in a real-time, prioritized and secure manner.
Cost-effective expansion of existing log analytics infrastructure by improving error detection and troubleshooting through more comprehensive data sets.
Intelligent edge analytics to support real-time content-based routing, prioritization, and simultaneous delivery of data into Connected Data Platforms, log analytics and reporting systems for comprehensive coverage and retention of Internet of Anything data.

Published in: Technology
  • Be the first to comment

Log Analytics Optimization

  1. 1. Optimizing Log Analytics from the Edge April 2016 © Hortonworks Inc. 2011 – 2015. All Rights Reserved
  2. 2. 2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved About Hortonworks Customer Momentum ~800 customers (as of Feb 10, 2016) Publicly traded on NASDAQ: HDP Hortonworks Data Platform Completely open multi-tenant platform for any app and any data Consistent enterprise services for security, operations, and governance Partner for Customer Success Leader in open-source community, focused on innovation to meet enterprise needs Unrivaled Hadoop support subscriptions Founded in 2011 Original 24 architects, developers, operators of Hadoop from Yahoo! 800+ E M P L O Y E E S 1500+ E C O S Y S T E M PA R T N E R S
  3. 3. 3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved EMBRACE AN OPEN APPROACH MASTER THE VALUE OF DATA EVERY BUSINESS IS A DATA BUSINESS
  4. 4. 4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved DATA AT REST DATA IN MOTION ACTIONABLE INTELLIGENCE MODERN DATA APPLICATIONS Actionable Intelligence from Connected Data Platforms Capturing perishable insights from data in motion Ensuring rich, historical insights on data at rest Necessary for modern data applications Hortonworks DataFlow Hortonworks Data Platform
  5. 5. 5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Optimizing Log Ingest with Hortonworks DataFlow
  6. 6. 6 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Why Hortonworks DataFlow? Because even the best data scientists and most powerful platforms need the right data to analyze
  7. 7. 7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Store Data Process and Analyze Data Acquire Data Perception of DataFlows: Easy, Definitive Dataflow
  8. 8. 8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Reality of Dataflows: Complex, Convoluted Store Data Process and Analyze Data Acquire Data Store DataStore Data Store Data Store Data Acquire Data Acquire Data Acquire Data Dataflow
  9. 9. 9 © Hortonworks Inc. 2011 – 2016. All Rights Reserved HDF has 130+ Processors - Multiple for Log Analytics HTTP Syslog Email HTML Image Hash Encrypt Extract TailMerge Evaluate Duplicate Execute Scan GeoEnrich Replace ConvertSplit Translate HL7 FTP UDP XML SFTP Route Content Route Context Route Text Control Rate Distribute Load AMQP
  10. 10. 10 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Log Analytics Systems Today LOG ANALYTICS PLATFORMNetwork Device Logs • Not all data can be captured • Not all captured data is valuable • Transport all data
  11. 11. 11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Cost Effectively Expand Storage Options of Log Data LOG ANALYTICS PLATFORM Network Device Logs HDP HDF 3. Cost effectively expand collection and grow timescale of logs collected 2. Content-based routing based on dynamic evaluation of content, attributes, priority 1. Integrate and enrich logs across data centers and security zones
  12. 12. 12 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Efficiently Expand Log Ingestion from the Edge LOG ANALYTICS PLATFORM Network Device Logs HDF HDF HDF HDPHDF • Expand collection to new sources of machine data • Edge analytics to transform, enrich and prioritize content based routing • Capture and transport only valuable data
  13. 13. 13 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Expand Analytics and Reporting Options with HDP LOG ANALYTICS PLATFORM Network Device Logs HDF HDF HDF HDPHDF ODBC interface traditional BI tools Easy access to log analytics data through traditional BI tools Give data scientists better tooling – Spark, Storm etc
  14. 14. 14 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Expand to small scale, remote systems LOG ANALYTICS PLATFORM Network Device Logs HDF HDF HDF HDPHDF
  15. 15. 15 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Optimize Log Analytics with Content Based Routing LOG ANALYTICS PLATFORM Edge analytics for cost-effective and efficient movement of machine data HDF Intelligent, content based routing, transformation and enrichment Send data to alternative systems based on value, content, priority HDP HDF HDF HDF
  16. 16. 16 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Splunk Optimization: Using HDP as Data Refinery
  17. 17. 17 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Splunk Hadoop Connect 17  Reliable bi-directional integration Import Browse Export Splunk Hadoop Connect >2000 downloads HA Indexes and Storage Commodity Servers Hadoop (MapReduce & HDFS) Report & analyze Custom dashboards Monitor and alert Ad hoc search
  18. 18. 18 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Splunk, Hunk & Hortonworks YARN Ready Partner Certified on Hortonworks Data Platform Existing Sandbox tutorial
  19. 19. 19 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Splunk, Part of the Modern Data Architecture • Bi-directional data integration between Splunk & HDP • Collect data from across the organization, deliver it to Hadoop for refining data and batch analytics • Output of Hadoop jobs can be imported into Splunk Enterprise for rapid analysis and visualization • Archiving from Splunk Enterprise to Hadoop
  20. 20. 20 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Splunk, Part of the Modern Data Architecture • Bi-directional data integration between Splunk & HDP • Collect data from across the organization, deliver it to Hadoop for refining data and batch analytics • Output of Hadoop jobs can be imported into Splunk Enterprise for rapid analysis and visualization • Archiving from Splunk Enterprise to Hadoop
  21. 21. 21 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Hunk + Hortonworks 21 Explore, analyze and visualize data in HDP from one integrated platform Simply point Hunk at your HDP cluster(s) and start exploring data immediately Search data, change perspectives and preview results as MapReduce jobs run INTERACTIVE EXPLORATION RICH DEVELOPER ENVIRONMENT Build big data apps on data in HDP using standard web languages and frameworks FULL-FEATURED ANALYTICS FAST TO DEPLOY AND DRIVE VALUE
  22. 22. 22 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Augment Splunk Deployment with Hortonworks Data Platform Heavy Indexer Universal Forwarders HDP Enables Splunk Storage • Expansion to more data than previously feasible • Archive data from Splunk into Hadoop • Query archived Splunk data in Hadoop • Focus Splunk infrastructure on what really matters
  23. 23. 23 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Find out how much you can optimize your log analytics infrastructure today. Contact sales@hortonworks.com

×