Copyright © 2013 Splunk Inc.Raanan DaganMay 2013Splunk Big DataArchitectural Patterns:Hadoop and Database
Agenda• Architectural Patterns - Splunk and Hadoop• Hadoop Connect Demo• Architectural Patterns - Splunk and Database• DB ...
3DeveloperPlatformReportandanalyzeCustomdashboardsMonitorand alertAd hocsearchSQLSplunk Hadoop Connect• Reliable bi-direct...
Architectural PatternsSplunk and Hadoop
5Splunk Hadoop ConnectDelivers reliable integrationbetween Splunk and HadoopExport events to HadoopExplore and Browse Hado...
Hadoop Connect: 3 PatternsSplunk thenHadoop1Hadoop thenSplunkCombination andSearchCommands2 36
Splunk then HadoopSplunk Ingest and provides: Real-time Analytics, End-userSecurity and Visualization7Splunk forAnalyticsD...
Hadoop Connect - Export81. Splunk forwarders move data to anindexer2. Search head stream data into a localdirectory3. Peri...
Hadoop Connect: 3 Patterns9Splunk thenHadoop1Hadoop thenSplunkCombination andSearchCommands2 3
Hadoop then Splunk10Splunk forAnalyticsData Sources Hadoop forETLHadoop Ingest. Splunk provides InteractiveAnalytics, End-...
Hadoop Connect – Import and Index111. Splunk detects any updated or new filein the HDFS directory2. Splunk imports the dat...
Hadoop Connect: 3 Patterns12Splunk thenHadoop1Hadoop thenSplunkCombination andSearchCommands2 3
Splunk forAnalyticsData SourcesHadoop forETL• Splunk and Hadoop Sharethe data• Splunk for real-timeAnalytics• Hadoop for E...
Hadoop Connect – ExploreEnables Splunk to browse and navigateHDFS directories and files from the Splunksearch head user in...
Splunk Hadoop ConnectDemo
Architectural PatternsSplunk and Database
Splunk DB ConnectEnrich search results with additionalbusiness contextEasily import data into Splunk fordeeper analysisInt...
DB Connect: 3 Patterns18Database Lookup1Import DatabaseTablesSearchCommands2 3
19Media ServerLogs(Machine Data)Mar 01 19:18:50:000 aaa2 radiusd[12548]:[ID 959576 local1.info] INFO RADOP(13) acct start ...
DB Connect – Database Lookup201. Use ‘lookup’ in Splunk Search2. Keys are sent from Splunk to Databases (Product ID)3. Val...
DB Connect: 3 Patterns21Database Lookup1Import DatabaseTablesSearchCommands2 3
DB Connect - Import Database Table22Provide deeper analysisMachineDataMachineDataMachineDataRDBMS
DB Connect – Import Database Table23Two input types can be used to import rows from the database:• Tail = Bring only new o...
DB Connect: 3 Patterns24Database Lookup1Import DatabaseTablesSearchCommands2 3
Splunk Search Language ExtensionsExecute database queries directly from the Splunk user interface withnew Dbquery and Dbin...
DB Connect – Search Commands261. Use SQL-92 or Stored Procedures with DBQuery2. Database Info user interface wraps DBinfo ...
Splunk DB ConnectDemo27
Summary28Splunk and Hadoop:– Splunk provides real-time analysis, visualization, and security– Hadoop provides parallel ETL...
QuestionsRaanan Daganrdagan@splunk.com
Upcoming SlideShare
Loading in …5
×

SplunkLive! Washington DC May 2013 - Big Data Architectural Patterns

1,529 views

Published on

Published in: Technology
0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,529
On SlideShare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
0
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide
  • Splunk Hadoop Connect has three main functions:Export events to Hadoop – Collect and index massive streams of machine data in real-time using Splunk. Search, analyze and visualize your data and provide secure dashboards to multiple users across your organization. Then send events in a reliable, predictable way to HDFS for archiving, further processing or additional batch analytics. Users have several options when exporting events. They can pre-process data in Splunk before exporting the results into Hadoop, or they can simply export raw events. For example, when processing Apache® web logs, you can limit the data export to just the client IP, session ID and the URL fields. Explore Hadoop directories and files – Browse and navigate HDFS directories and files from the Splunk Hadoop Connect user interface, before deciding to import data into Splunk. View directories, files names, permissions, sizes and dates of the last modification, all without having to open the actual files. Drill down into a set of directories, examine files and with a click of a button import and index data in Splunk. Import and Index Hadoop data in Splunk – Address Hadoop limitations, such as the lack of visualizations, limited support for user access controls, and the need for data scientists and specialists to analyze data or to write MapReduce code. Splunk Hadoop Connect detects any updated or new file in the HDFS directory, for example as a result of Sqoop, Hbase or Hadoop commands, and collects and indexes the data. Once in Splunk, your data is immediately available for searching, reporting, analysis and visualizations and protected by role-based access controls.
  • There are a whole host of ways Developerscan leverage Splunk to maximize enterprise technology investments.Accelerate Dev & Test: Use Splunk Enterprise out of the box. Splunk increases the speed and efficiency of application development, testing and provides proactive monitoring and analytics for applications in production.Integrate with IT Infrastructure: Integrate Splunk data with other enterprise applications, using SDKs on top of our REST API. Build real-time data applications: Build applications that take the value of Splunk beyond IT. IT early-warning systems, security and fraud protection, clickstream analysis & other revenue enhancing analytics.
  • There are a whole host of ways Developerscan leverage Splunk to maximize enterprise technology investments.Accelerate Dev & Test: Use Splunk Enterprise out of the box. Splunk increases the speed and efficiency of application development, testing and provides proactive monitoring and analytics for applications in production.Integrate with IT Infrastructure: Integrate Splunk data with other enterprise applications, using SDKs on top of our REST API. Build real-time data applications: Build applications that take the value of Splunk beyond IT. IT early-warning systems, security and fraud protection, clickstream analysis & other revenue enhancing analytics.
  • There are a whole host of ways Developerscan leverage Splunk to maximize enterprise technology investments.Accelerate Dev & Test: Use Splunk Enterprise out of the box. Splunk increases the speed and efficiency of application development, testing and provides proactive monitoring and analytics for applications in production.Integrate with IT Infrastructure: Integrate Splunk data with other enterprise applications, using SDKs on top of our REST API. Build real-time data applications: Build applications that take the value of Splunk beyond IT. IT early-warning systems, security and fraud protection, clickstream analysis & other revenue enhancing analytics.
  • Splunk DB Connect delivers reliable, scalable, real-time integration between Splunk Enterprise and traditional relational databases. With Splunk DB Connect, structured data from relational databases can be easily integrated into Splunk Enterprise, driving deeper levels of operational intelligence and richer business analytics across the organization.Organizations can drive more meaningful insights for IT operations, security and business users. For example, IT operations teams can track performance, outage and usage by department, location and business entities. Security professionals can correlate machine data with critical assets and watch-lists for: incident investigations, real-time correlations and advanced threat detection using the award-winning Splunk Enterprise. Business users can analyze service levels and user experience by customer in real-time to make more informed decisions.
  • There are a whole host of ways Developerscan leverage Splunk to maximize enterprise technology investments.Accelerate Dev & Test: Use Splunk Enterprise out of the box. Splunk increases the speed and efficiency of application development, testing and provides proactive monitoring and analytics for applications in production.Integrate with IT Infrastructure: Integrate Splunk data with other enterprise applications, using SDKs on top of our REST API. Build real-time data applications: Build applications that take the value of Splunk beyond IT. IT early-warning systems, security and fraud protection, clickstream analysis & other revenue enhancing analytics.
  • None of the customer's identifying information is in the logs (which I will add to the dashboard); we simply identified a phone that has had problems downloading music, and enriched that information with data from our customers database.
  • There are a whole host of ways Developerscan leverage Splunk to maximize enterprise technology investments.Accelerate Dev & Test: Use Splunk Enterprise out of the box. Splunk increases the speed and efficiency of application development, testing and provides proactive monitoring and analytics for applications in production.Integrate with IT Infrastructure: Integrate Splunk data with other enterprise applications, using SDKs on top of our REST API. Build real-time data applications: Build applications that take the value of Splunk beyond IT. IT early-warning systems, security and fraud protection, clickstream analysis & other revenue enhancing analytics.
  • This leading national telecommunications company in Canada, with $10.8 billion of annual revenue and 13.0 million customer connections including 7.6 million wireless subscribers, 3.4 million wireline network access lines, 1.3 million Internet subscribers and more than 635,000 TV customers.For their use case, they are collecting all network switch data into a single Oracle Database. Oracle tables stores: CPU utilization per switch / inbound pockets per switch / 30 different stats going to Oracle Database. In addition, they have an alerts system when there is a high utilization of network traffic.Splunk DB Connect is used to import the data from Oracle for more effective analysis:Detect high utilization patterns. Trending behavior in a specific time of the day per switch (is it normal or not). Visualizing outliers (high performance patterns).With Splunk the team can take proactive measures before the network is saturated, which reduce network utilization alerts. Volume = Monitoring over 10,000 switches, each with multiple interfaces. Splunk imports and indexesmillions of rows per day.It took the main Splunk user only 2 days from initial installation to graph network trends / outliers.
  • There are a whole host of ways Developerscan leverage Splunk to maximize enterprise technology investments.Accelerate Dev & Test: Use Splunk Enterprise out of the box. Splunk increases the speed and efficiency of application development, testing and provides proactive monitoring and analytics for applications in production.Integrate with IT Infrastructure: Integrate Splunk data with other enterprise applications, using SDKs on top of our REST API. Build real-time data applications: Build applications that take the value of Splunk beyond IT. IT early-warning systems, security and fraud protection, clickstream analysis & other revenue enhancing analytics.
  • Splunk software lets users search and navigate their data from one place. Splunk DB Connect includes search language extensions that can be executed directly from the Splunk user interface. Dbquery and Dbinfo are Splunk search commands that enable you to execute database queries directly from the Splunk Enterprise user interface. Dbinfo fetches schema information from the database.
  • Quick to set-up, scales to multiple concurrent databasesEnrich machine data with structured data from relational databasesExecute database queries directly from the Splunk user interface Browse and navigate database schemas and tablesCombine machine data with structured data from relational databases
  • SplunkLive! Washington DC May 2013 - Big Data Architectural Patterns

    1. 1. Copyright © 2013 Splunk Inc.Raanan DaganMay 2013Splunk Big DataArchitectural Patterns:Hadoop and Database
    2. 2. Agenda• Architectural Patterns - Splunk and Hadoop• Hadoop Connect Demo• Architectural Patterns - Splunk and Database• DB Connect Demo
    3. 3. 3DeveloperPlatformReportandanalyzeCustomdashboardsMonitorand alertAd hocsearchSQLSplunk Hadoop Connect• Reliable bi-directionalintegration to HadoopSplunk DB Connect• Real-time integrationto relational DBsSplunk: A Platform for Big Data Integration
    4. 4. Architectural PatternsSplunk and Hadoop
    5. 5. 5Splunk Hadoop ConnectDelivers reliable integrationbetween Splunk and HadoopExport events to HadoopExplore and Browse HadoopdirectoriesImport and Index Hadoop datainto Splunk
    6. 6. Hadoop Connect: 3 PatternsSplunk thenHadoop1Hadoop thenSplunkCombination andSearchCommands2 36
    7. 7. Splunk then HadoopSplunk Ingest and provides: Real-time Analytics, End-userSecurity and Visualization7Splunk forAnalyticsData Sources Hadoop forETL
    8. 8. Hadoop Connect - Export81. Splunk forwarders move data to anindexer2. Search head stream data into a localdirectory3. Periodically Splunk compresses the fileand puts it into the HDFS directory(location set by users)
    9. 9. Hadoop Connect: 3 Patterns9Splunk thenHadoop1Hadoop thenSplunkCombination andSearchCommands2 3
    10. 10. Hadoop then Splunk10Splunk forAnalyticsData Sources Hadoop forETLHadoop Ingest. Splunk provides InteractiveAnalytics, End-user Security, and Visualization
    11. 11. Hadoop Connect – Import and Index111. Splunk detects any updated or new filein the HDFS directory2. Splunk imports the data into Splunkindexers3. In Splunk you can apply access controlsto the data as well as search, report andvisualize your data
    12. 12. Hadoop Connect: 3 Patterns12Splunk thenHadoop1Hadoop thenSplunkCombination andSearchCommands2 3
    13. 13. Splunk forAnalyticsData SourcesHadoop forETL• Splunk and Hadoop Sharethe data• Splunk for real-timeAnalytics• Hadoop for ETLData flows in both directions13
    14. 14. Hadoop Connect – ExploreEnables Splunk to browse and navigateHDFS directories and files from the Splunksearch head user interfaceExplore User interface wraps ‘hdfs lsr’ and‘hdfs read’14| hdfs read hdfs://kiru-demo-01.sv.splunk.com:9000/home/HadoopConnect/twitter/File1.gz| hdfs lsr hdfs://kiru-demo-01.sv.splunk.com:9000/home/HadoopConnect/twitter/
    15. 15. Splunk Hadoop ConnectDemo
    16. 16. Architectural PatternsSplunk and Database
    17. 17. Splunk DB ConnectEnrich search results with additionalbusiness contextEasily import data into Splunk fordeeper analysisIntegrate multiple DBs concurrentlySimple set-up, non-evasive and secureReliable, scalable, real-timeintegration between Splunk andtraditional relational databasesMicrosoft SQLServerJDBCDatabaseLookupDatabaseQueryConnectionPoolingOtherDatabasesOracleDatabaseJava Bridge Server17
    18. 18. DB Connect: 3 Patterns18Database Lookup1Import DatabaseTablesSearchCommands2 3
    19. 19. 19Media ServerLogs(Machine Data)Mar 01 19:18:50:000 aaa2 radiusd[12548]:[ID 959576 local1.info] INFO RADOP(13) acct start for 2172618992@splunktel.com10.164.232.181 from 12.130.60.5 recorded OK.2013-03-01 19:18:50:150 10.2.1.34 GET /sync/addtolibrary/01011207201000005652000000000053 - 80 - 10.164.232.181 "Mozilla/5.0(iPhone; CPU iPhone OS 5_0_1 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) Version/5.1 Mobile/9A405 Safari/7534.48.3"503 0 0 825 1680Mar 01 19:18:50:163 aaa2 radiusd[12548]:[ID 959576 local1.info] INFO RADOP(13) acct stop for 2172618992@splunktel.com10.164.232.181 from 12.130.60.5 recorded OK.Track ID Artist Title Format ID Run time01011207201000005652000000000053 Maroon 5 Moves like Jagger MP3 4:30Phone # Subscriber ID2172618992 53546SubscriberIDFirst Name Last Name Age State CustomerScore53546 Jim Morrison 25 CA 93Customer,ProductDatabasesPhone Number IP Address Track IDEnrich Machine Data with additional Business ContextDB Connect – Database Lookup
    20. 20. DB Connect – Database Lookup201. Use ‘lookup’ in Splunk Search2. Keys are sent from Splunk to Databases (Product ID)3. Values are returned if Keys are matched (Product Name, Product Price)SQLKEYSVALUES
    21. 21. DB Connect: 3 Patterns21Database Lookup1Import DatabaseTablesSearchCommands2 3
    22. 22. DB Connect - Import Database Table22Provide deeper analysisMachineDataMachineDataMachineDataRDBMS
    23. 23. DB Connect – Import Database Table23Two input types can be used to import rows from the database:• Tail = Bring only new or updated rows• Dump = Bring entire tableSQLImport
    24. 24. DB Connect: 3 Patterns24Database Lookup1Import DatabaseTablesSearchCommands2 3
    25. 25. Splunk Search Language ExtensionsExecute database queries directly from the Splunk user interface withnew Dbquery and Dbinfo Splunk search commands25*** DBoutput (BETA) - Create or Update database records on information Splunk searches
    26. 26. DB Connect – Search Commands261. Use SQL-92 or Stored Procedures with DBQuery2. Database Info user interface wraps DBinfo and DBquery commandsSQLQUERYRESULTS
    27. 27. Splunk DB ConnectDemo27
    28. 28. Summary28Splunk and Hadoop:– Splunk provides real-time analysis, visualization, and security– Hadoop provides parallel ETL or batch computationSplunk and Database:– Enrich search results with additional business context– Import data into Splunk for deeper analysis
    29. 29. QuestionsRaanan Daganrdagan@splunk.com

    ×