Your SlideShare is downloading. ×
Copyright © 2013 Splunk Inc.Tapan Bhatt / Raanan DaganMay 2013Splunk DB Connect:Enrich Machine Data withStructured Data
AgendaBackground and OverviewDB Connect DemoTechnical OverviewCustomer Examples & SummaryQuestions
Splunk: the Platform for Machine Data3Real-time Business InsightsOperational VisibilityProactive MonitoringSearch and Inve...
What about Structured Data?4CustomerProfileProductAttributesEmployeeDetailsPricing &Rate PlansAssetInfo
Machine Data – Delivers Real-time Insights5Media ServerLogs(Machine Data)Mar 01 19:18:50:000 aaa2 radiusd[12548]:[ID 95957...
Structured Data – Contains Business Context6Media ServerLogs(Machine Data)Mar 01 19:18:50:000 aaa2 radiusd[12548]:[ID 9595...
Operational Dashboards with Business Context7Top TracksUser ActivityClick to investigateCustomer experience Download Error...
Enrich Machine Data with Structured Data8Structured DatabasesCSV LookupDB ConnectLaunched March 2013
Introducing Splunk DB ConnectEnrich search results with additionalbusiness contextEasily import data into Splunk fordeeper...
Delivering Operational Intelligence10IT Operations Analytics> Machine DataApplicationlogs, monitoring data,disk utilizatio...
Splunk DB ConnectDemo11
Splunk DB ConnectTechnical Overview
Splunk DB Connect: Main FeaturesDatabase Connection ManagementSQL Database LookupsSplunk Search Language extensions– Datab...
Installing Splunk DB ConnectSimple app setup, no configuration files to touchAutomatically checks for the required Java ve...
Database Connection ManagementConfigure new database connection settings in minutesfrom the Splunk user interface15
Microsoft SQLServerJDBCDatabaseLookupDatabaseQueryConnectionPoolingOtherDatabasesOracleDatabaseJava Bridge ServerWorks wit...
Database Lookups17Enrich machine data by adding structured data fromtraditional relational databases
Three Steps to Enriching Machine Data181. Connect2. Configure3. Enrich
Splunk Search Language ExtensionsExecute database queries directly from the Splunk user interface withnew Dbquery and Dbin...
Explore Database StructureWrappingdbinfo anddbquery20Browse and navigate database schemas and tables from theSplunk DB Con...
Import and Index Database DataCombine machine data with structured data from relational databases21New dbmon-tail and dbmo...
Technical SummaryQuick to set-up, scales to multiple concurrent databasesEnrich machine data with database data in three e...
Success Stories
Powering Search Analytics24Understanding customerusageClient Name,Country, EmailFeed IDSQL SQL SQL= Client Databases conta...
Enabling Exceptional Customer Service25Users to CustomersmappingSQL SQLUserActivity= Customer details,external/internal de...
Powerful Connectivity Drives Better Insights26DeveloperPlatformReportandanalyzeCustomdashboardsMonitorand alertAd hocsearc...
SummaryMachine data contains a categorical record of activity andbehaviorEnrich with structured data to provide business c...
QuestionsTapan Bhatttbhatt@splunk.comRaanan Daganrdagan@splunk.com
Upcoming SlideShare
Loading in...5
×

SplunkLive! New York April 2013 - Enrich Machine Data with Structured Data

2,288

Published on

Published in: Technology
0 Comments
5 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,288
On Slideshare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
0
Comments
0
Likes
5
Embeds 0
No embeds

No notes for slide
  • Splunk Enterprise is the platform for machine data. It reliably collects and indexes all the streaming data from IT systems and technology devices in real-time - tens of thousands of sources in unpredictable formats and types. Splunk software is optimized for real-time, low latency and interactivity.Once in Splunk, your data is available for searching, monitoring, analysis and visualizations. With Splunk you can gain new levels of visibility and insight. This is called Operational Intelligence.Use Splunk to:1. Find and fix problems dramatically faster2. Automatically monitor to identify issues, problems and attacks3. Gain end-to-end visibility to track and deliver on IT KPIs and make better-informed IT decisions4. Gain real-time insight from operational data to make better-informed business decisions
  • Splunk software has proven uses for IT, security and business users.These users can meaningfully improve their performance in a wide range of areas e.g. meet service levels, reduce costs, mitigate security risks, maintain compliance and gain insights.
  • None of the customer's identifying information is in the logs (which I will add to the dashboard); we simply identified a phone that has had problems downloading music, and enriched that information with data from our customers database.
  • None of the customer's identifying information is in the logs (which I will add to the dashboard); we simply identified a phone that has had problems downloading music, and enriched that information with data from our customers database.
  • This dashboard provides an example of the value of combining machine data with structured data. Top left: unique user activity can be gained directly from the machine data logs.Right and bottom: combining with structured data makes our dashboard more meaningful:Top track titles in aggregate (and updated in real-time) – can also be used to reconcile royalty payments in this example. Download errors by a specific device (phone number) Enrich behavior - click to bring up the customer record details and all associated activity for further analysis.
  • DB Connect available today – Steve Sommer and I met with CIO of Rakuten, a leading online retailer from Japan last year. Nice guy, very interested in Splunk but then he said, I want correlate all this data with my product master. Ok, how large is your product master? We have 3 millions SKUs….no way we can handle with CSV look up. But now, we can handle with DB Connect – in real-time, at scale with much easier set up & maintenance.Guys, this is a huge Game Changer….
  • Splunk DB Connect delivers reliable, scalable, real-time integration between Splunk Enterprise and traditional relational databases. With Splunk DB Connect, structured data from relational databases can be easily integrated into Splunk Enterprise, driving deeper levels of operational intelligence and richer business analytics across the organization.Organizations can drive more meaningful insights for IT operations, security and business users. For example, IT operations teams can track performance, outage and usage by department, location and business entities. Security professionals can correlate machine data with critical assets and watch-lists for: incident investigations, real-time correlations and advanced threat detection using the award-winning Splunk Enterprise. Business users can analyze service levels and user experience by customer in real-time to make more informed decisions.
  • To net it out, what DB Connect does is provide business context to machine data. Sales Analytics – activation data + rate plan + customer profileProduct Analytics – application logs + customer profile (what features are being used by what type of customers and when – in real-time)We talk about how with Splunk, all you need a browser and your imagination. With DB Connect, you need to take that imagination to the next level because so many amazing use cases are now at your finger tips!Learn more about this at the showcase…..
  • Splunk DB Connect is simple to install and set up. Simply go to Splunkbase and download the App. It automatically checks for the required Java version.
  • UI-driven, users can configure new databases in minutes.
  • Splunk DB Connect is compatible with most relational databases including Oracle® Database, Microsoft® SQL Server, Sybase®, PostgreSQL, MySQL™, SQLite, H2, HyperSQL and support for Generic ODBC driver.
  • With Database Lookup, users can enrich machine-generated data by adding structured data from relational databases. For example, by using Splunk Enterprise and Splunk DB Connect, key values contained in machine data can be used to reference related business data in relational databases, such as device addresses, product codes, media identifiers, etc. For example, telecom providers have the ability to combine real-time service activation data with profile data from a customer master database to understand what types of customers are purchasing what types of plans – enabling in-depth real-time sales and customer analytics not possible before.
  • To summarize, we’ve made it easy to connect to new relational databases and start enriching machine data.With Splunk DB Connect, the addition of business context from structured sources delivers new value and actionable metrics for any organization.
  • Splunk software lets users search and navigate their data from one place. Splunk DB Connect includes search language extensions that can be executed directly from the Splunk user interface. Dbquery and Dbinfo are Splunk search commands that enable you to execute database queries directly from the Splunk Enterprise user interface. Dbinfo fetches schema information from the database.
  • Splunk software lets users search and navigate their data from one place. Splunk DB Connect includes search language extensions that can be executed directly from the Splunk user interface. Dbquery performs SQL queries and presents the results as Splunk visualizations. For example, dbquery database=ASSETDB “SELECT hostname, owner, department from host_information WHERE location LIKE ‘%NY%.
  • Import and Index Data from Relational Databases into Splunk. Combine business/structured data from relational databases with machine data to drive end-to-end operational insights. The Splunk Tail command can be use to detect updated or new rows in the database by referencing time stamp values. Splunk DB Connect also enables you to import data via periodic snapshots of the database—where database tables are recorded from a single point in time.
  • Quick to set-up, scales to multiple concurrent databasesEnrich machine data with structured data from relational databasesExecute database queries directly from the Splunk user interface Browse and navigate database schemas and tablesCombine machine data with structured data from relational databases
  • Community search engine Boardreader uses Splunk software to monitor its web crawlers and other essential business systems. In addition to generating large amounts of machine data on a daily basis, Boardreader maintains SQL databases to store business data and lookup tables such as country names, languages, production metrics and status, event and error codes. With Splunk DB Connect, Boardreader can run internal reports on usage trends and enhance the queries and reports with data from their SQL databases. Boardreader says Splunk DB Connect enabled it to replace its data warehouse.Boardreader offer two main services:Website – can be used anonymously or using a login for access and use certain features or functions, such as receiving email alerts, or certain services, usersneed to register and create an account.APIs – used by business customers to to run queries and load their analytical systems with critical social media data. 100s of requests per second.Before Splunk:Their existing reporting was broken. Too much data (and growing).Visibility was at best “a day behind”.“Online services, require real-time analytics”.Finding Splunk:In minutes: Downloaded Splunk and blasted it with packets. And it didn’t even break a sweat.In hours: Showed our CEO a dashboard – when he saw the graphics building in real-time and knew the data volumes behind it, he wanted it.In days: Entire Splunk deployment up and running and generating reports and dashboards.Now have processed and analyzed billions of events in Splunk.In terms of their environment:As well as data from 1000s of servers, they are bringing together business and IT data from 3 Microsoft SQL Server & MySQL databases (15-20 eventually possible) for real-time monitoring and some real-time business analytics.Standout factor of Splunk: Real-time. They are basically replacing a data warehouse.
  • Corporation Service Company (CSC) is a leader for business, legal and financial services worldwide, offering a solution for every phase of the business life cycle. CSC uses Splunk Enterprise to monitor and correlate page load, traffic and user data to help deliver exceptional customer service (something they pride themselves on). Splunk DB Connect makes it easier for CSC customer service representatives to see top usage, overall volume and any problems customers are facing, by ensuring that they can map the customers’ usage and experience from the machine logs with their credentials contained in the database.CSC use the User ID from the machine data to correlate and analyze usage with customer details from their customer master.Before DB Connect, CSC used Splunk normal Lookup with CSV files. The main problem was that these CSV files where not refreshed for 90 days. So although the customer added new users to the database, these new users where not reflected in the CSV file. Therefore, the monitoring and reports included inaccurate Customer data.
  • Most organizations maintain a diverse set of data stores – machine data, relational data and other unstructured data. Splunk DB Connect delivers real-time connectivity to relational databases and Splunk Hadoop Connect delivers bi-directional connectivity to Hadoop. Both Splunk Apps enable you to drive more meaningful insights from all of your data.
  • Transcript of "SplunkLive! New York April 2013 - Enrich Machine Data with Structured Data"

    1. 1. Copyright © 2013 Splunk Inc.Tapan Bhatt / Raanan DaganMay 2013Splunk DB Connect:Enrich Machine Data withStructured Data
    2. 2. AgendaBackground and OverviewDB Connect DemoTechnical OverviewCustomer Examples & SummaryQuestions
    3. 3. Splunk: the Platform for Machine Data3Real-time Business InsightsOperational VisibilityProactive MonitoringSearch and InvestigationOperational IntelligenceMachine Data
    4. 4. What about Structured Data?4CustomerProfileProductAttributesEmployeeDetailsPricing &Rate PlansAssetInfo
    5. 5. Machine Data – Delivers Real-time Insights5Media ServerLogs(Machine Data)Mar 01 19:18:50:000 aaa2 radiusd[12548]:[ID 959576 local1.info] INFO RADOP(13) acct start for 2172618992@splunktel.com10.164.232.181 from 12.130.60.5 recorded OK.2013-03-01 19:18:50:150 10.2.1.34 GET /sync/addtolibrary/01011207201000005652000000000053 - 80 - 10.164.232.181 "Mozilla/5.0(iPhone; CPU iPhone OS 5_0_1 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) Version/5.1 Mobile/9A405 Safari/7534.48.3"503 0 0 825 1680Mar 01 19:18:50:163 aaa2 radiusd[12548]:[ID 959576 local1.info] INFO RADOP(13) acct stop for 2172618992@splunktel.com10.164.232.181 from 12.130.60.5 recorded OK.Phone Number IP AddressTrack ID
    6. 6. Structured Data – Contains Business Context6Media ServerLogs(Machine Data)Mar 01 19:18:50:000 aaa2 radiusd[12548]:[ID 959576 local1.info] INFO RADOP(13) acct start for 2172618992@splunktel.com10.164.232.181 from 12.130.60.5 recorded OK.2013-03-01 19:18:50:150 10.2.1.34 GET /sync/addtolibrary/01011207201000005652000000000053 - 80 - 10.164.232.181 "Mozilla/5.0(iPhone; CPU iPhone OS 5_0_1 like Mac OS X) AppleWebKit/534.46 (KHTML, like Gecko) Version/5.1 Mobile/9A405 Safari/7534.48.3"503 0 0 825 1680Mar 01 19:18:50:163 aaa2 radiusd[12548]:[ID 959576 local1.info] INFO RADOP(13) acct stop for 2172618992@splunktel.com10.164.232.181 from 12.130.60.5 recorded OK.Track ID Artist Title Format ID Run time01011207201000005652000000000053 Maroon 5 Moves like Jagger MP3 4:30Phone # Subscriber ID2172618992 53546SubscriberIDFirst Name Last Name Age State CustomerScore53546 Jim Morrison 25 CA 93Customer,ProductDatabasesPhone Number IP Address Track ID
    7. 7. Operational Dashboards with Business Context7Top TracksUser ActivityClick to investigateCustomer experience Download Errorsby device
    8. 8. Enrich Machine Data with Structured Data8Structured DatabasesCSV LookupDB ConnectLaunched March 2013
    9. 9. Introducing Splunk DB ConnectEnrich search results with additionalbusiness contextEasily import data into Splunk fordeeper analysisIntegrate multiple DBs concurrentlySimple set-up, non-evasive and secureReliable, scalable, real-timeintegration between Splunk andtraditional relational databasesMicrosoft SQLServerJDBCDatabaseLookupDatabaseQueryConnectionPoolingOtherDatabasesOracleDatabaseJava Bridge Server9
    10. 10. Delivering Operational Intelligence10IT Operations Analytics> Machine DataApplicationlogs, monitoring data,disk utilizationOperational IntelligenceSecurity AnalyticsFirewall logs, Radiuslogs, Nessus vulnerabilityCritical assets, watch-lists, privileged user lists,black-lists, device data>CMDB, assetinventory, topology, user, cost and departmentinformationStructured DataBusiness AnalyticsDevice activation,Radius, application logsRate plans, customerprofile, geo location
    11. 11. Splunk DB ConnectDemo11
    12. 12. Splunk DB ConnectTechnical Overview
    13. 13. Splunk DB Connect: Main FeaturesDatabase Connection ManagementSQL Database LookupsSplunk Search Language extensions– Database Query– Database InfoSQL Database Input13
    14. 14. Installing Splunk DB ConnectSimple app setup, no configuration files to touchAutomatically checks for the required Java version14
    15. 15. Database Connection ManagementConfigure new database connection settings in minutesfrom the Splunk user interface15
    16. 16. Microsoft SQLServerJDBCDatabaseLookupDatabaseQueryConnectionPoolingOtherDatabasesOracleDatabaseJava Bridge ServerWorks with Many DatabasesSupports mainstream databases– Oracle Database– Microsoft SQL Server– MySQL– PostgreSQL– Sybase– Generic JDBC supportDatabase connection pooling limitsload on Database16
    17. 17. Database Lookups17Enrich machine data by adding structured data fromtraditional relational databases
    18. 18. Three Steps to Enriching Machine Data181. Connect2. Configure3. Enrich
    19. 19. Splunk Search Language ExtensionsExecute database queries directly from the Splunk user interface withnew Dbquery and Dbinfo Splunk search commands19*** DBoutput (BETA) - Create or Update database records on information Splunk searches
    20. 20. Explore Database StructureWrappingdbinfo anddbquery20Browse and navigate database schemas and tables from theSplunk DB Connect user interface
    21. 21. Import and Index Database DataCombine machine data with structured data from relational databases21New dbmon-tail and dbmon-dumpinput types can be used to importrows from the database
    22. 22. Technical SummaryQuick to set-up, scales to multiple concurrent databasesEnrich machine data with database data in three easy stepsExecute SQL queries to visualize database data directly in theSplunk user interfaceImport and index database data for historical analysis andcorrelation with machine data22
    23. 23. Success Stories
    24. 24. Powering Search Analytics24Understanding customerusageClient Name,Country, EmailFeed IDSQL SQL SQL= Client Databases containClient Name, Country andEmail informationDatabase+MachineData= Search Activity tracked byFeed ID
    25. 25. Enabling Exceptional Customer Service25Users to CustomersmappingSQL SQLUserActivity= Customer details,external/internal detailsDatabase+MachineData= User activity data fromSaaS application, websitesSaaSReal-time visibility ofcustomer experienceWebsite
    26. 26. Powerful Connectivity Drives Better Insights26DeveloperPlatformReportandanalyzeCustomdashboardsMonitorand alertAd hocsearchSQLSplunk Hadoop Connect• Reliable bi-directionalintegration to HadoopSplunk DB Connect• Real-time integrationto relational DBsSplunk Dev Platform• API and SDKs to buildBig Data apps
    27. 27. SummaryMachine data contains a categorical record of activity andbehaviorEnrich with structured data to provide business context – forbetter IT, security and business insightsSplunk DB Connect delivers reliable, scalable, real-timeintegration between Splunk and traditional relational databases27
    28. 28. QuestionsTapan Bhatttbhatt@splunk.comRaanan Daganrdagan@splunk.com

    ×