Downloaded 42 times
![Operational IntelligenceAcrossthe Enterprise
IT professional
Create and share data models
Accelerate data models and custom
searches with the analytics store
Create reports with pivot
Developer Analyst
Leverage data models to
abstract data
Leverage pivot in custom apps
Create reports using pivot based on
data models created by IT
Pivot
Data
Model
Raw
Data
Analytics
Store
[10/11/12
18:57:04 UTC]
000000b0](https://image.slidesharecdn.com/datamodelspivotwithsplunkbreakoutsession-151015082147-lva1-app6891/75/Data-models-pivot-with-splunk-break-out-session-5-2048.jpg)
![Hurdles
• Simplesearcheseasy…Multi-stagemunging/reportingishard!
• Needtounderstanddata’sstructuretoconstructsearch
• Non-technicalusersmaynothavedatasourcedomainknowledge
• Splunkadminsdonothaveend-usersearchcontext
index=main source=*/banner_access* uri_path=/js/*/*/login/* guid=* useragent!=*KTXN* useragent!=*GomezAgent*
clientip!=206.80.3.67 clientip!=198.144.207.62 clientip!=97.65.63.66 clientip!=175.45.37.78
clientip!=209.119.210.194 clientip!=212.36.37.138 clientip!=204.156.84.0/24 clientip!=216.221.226.0/24
clientip!=207.87.200.162 | rex field=uri_path "/js/(?<t>[^/]*)/(?<v>[^/]*)/login/(?<l>[^/]*)” | eval license =
case(l LIKE "prod%" AND t="pro", "enterprise", l LIKE "trial%" AND t="pro", "trial", t="free", "free”) | rex field=v
"^(?<vers>d.d)” | bin span=1d _time as day | stats values(vers) as vers min(day) as min_day
min(eval(if(vers=="5.0", _time, null()))) as min_day_50 dc(day) as days values(license) as license by guid | eval
type = if(match(vers,"4.*"), "upgrade", "not upgrade") + "/" + if(days > 1, "repeat", "not repeat")| search
license=enterprise | eval _time = min_day_50| timechart count by type| streamstats sum(*) as *](https://image.slidesharecdn.com/datamodelspivotwithsplunkbreakoutsession-151015082147-lva1-app6891/75/Data-models-pivot-with-splunk-break-out-session-15-2048.jpg)
![Usingthe SplunkSearch Language
| pivot <modelname> <objectID> [statsfns, rowsplit, colsplit, filters, …]
Example:
| pivot WebIntelligence HTTP_Request count(HTTP_Request) AS "Count of HTTP_Request" SPLITROW
status AS "status" SORT 0 status
Behindthescenes:
sourcetype=access_* OR sourcetype=iis* uri=* uri_path=* status=* clientip=* referer=* useragent=*
| stats count AS "Count of HTTP_Request" by "status"
| sort limit=0 "status" | fields - _span
| fields "status", "Count of HTTP_Request"
| fillnull "Count of HTTP_Request"
| fields "status" *](https://image.slidesharecdn.com/datamodelspivotwithsplunkbreakoutsession-151015082147-lva1-app6891/75/Data-models-pivot-with-splunk-break-out-session-40-2048.jpg)
Uploaded byGeorg Knon
Data models pivot with splunk break out session
Here are the key points about data model acceleration in Splunk: - Data model acceleration optimizes searches that use data models by pre-processing constraints and attribute definitions at search time. This can significantly improve search performance. - Acceleration only applies to the first "event" object in the data model tree and its descendant objects. Searches against other object types like "search" or "transaction" do not benefit from acceleration. - The more filtering/extraction done in the data model objects, the more acceleration can improve performance by reducing the number of events earlier in the search pipeline. Simply defining fields may not yield huge gains. - Acceleration is most helpful for reports that run the same search repeatedly, like scheduled
More Related Content
Similar to Data models pivot with splunk break out session
Data models pivot with splunk break out session
- 1. Copyright © 2014Splunk Inc. DATA MODELS
- 2. During the courseof this presentation, we may make forward-looking statements regarding future events or the expected performance of the company. We caution you that such statements reflect our current expectations and estimates based on factors currently known to us and that actual events or results could differ materially. For important factors that may cause actual results to differ from those contained in our forward- looking statements, please review our filings with the SEC. The forward-looking statements made in this presentation are being made as of the time and date of its live presentation. If reviewed after its live presentation, this presentation may not contain current or accurate information. We do not assume any obligation to update any forward-looking statements we may make. In addition, any information about our roadmap outlines our general product direction and is subject to change at any time without notice. It is for informational purposes only and shall not, be incorporated into any contract or other commitment. Splunk undertakes no obligation either to develop the features or functionality described or to include any such feature or functionality in a future release. Splunk, Splunk>, Splunk Storm, Listen to Your Data, SPL and The Engine for Machine Data are trademarks and registered trademarks of Splunk Inc. in the United States and other countries. All other brand names, product names, or trademarks belong to their respective owners. ©2013 Splunk Inc. All rights reserved. Legal Notices 2
- 3.
- 4. AnalyticsBig Picture Build complexreports without the search language Provides more meaningful representation of underlying raw machine data Acceleration technology delivers up to 1000x faster analytics over Splunk 5 4 Pivot Data Model Analytics Store
- 5. Operational IntelligenceAcrossthe Enterprise ITprofessional Create and share data models Accelerate data models and custom searches with the analytics store Create reports with pivot Developer Analyst Leverage data models to abstract data Leverage pivot in custom apps Create reports using pivot based on data models created by IT Pivot Data Model Raw Data Analytics Store [10/11/12 18:57:04 UTC] 000000b0
- 6. Pivot is aquery builder.
- 7.
- 8.
- 9.
- 10.
- 11. Data set Business division Source Source Businessdivision Source Source
- 12. Common model Technology 1 Technology2 Technology 3
- 13.
- 14. sourcetype=access_combined source ="/home/ssorkin/banner_access.log.2013.6.gz" | eval unique=(uid + useragent) | stats dc(unique) by os_name | rename dc(unique) as "Unique Visitors" os_name as "Operating System" search and filter | munge | report | clean-up Splunk Search Language
- 15. Hurdles • Simplesearcheseasy…Multi-stagemunging/reportingishard! • Needtounderstanddata’sstructuretoconstructsearch •Non-technicalusersmaynothavedatasourcedomainknowledge • Splunkadminsdonothaveend-usersearchcontext index=main source=*/banner_access* uri_path=/js/*/*/login/* guid=* useragent!=*KTXN* useragent!=*GomezAgent* clientip!=206.80.3.67 clientip!=198.144.207.62 clientip!=97.65.63.66 clientip!=175.45.37.78 clientip!=209.119.210.194 clientip!=212.36.37.138 clientip!=204.156.84.0/24 clientip!=216.221.226.0/24 clientip!=207.87.200.162 | rex field=uri_path "/js/(?<t>[^/]*)/(?<v>[^/]*)/login/(?<l>[^/]*)” | eval license = case(l LIKE "prod%" AND t="pro", "enterprise", l LIKE "trial%" AND t="pro", "trial", t="free", "free”) | rex field=v "^(?<vers>d.d)” | bin span=1d _time as day | stats values(vers) as vers min(day) as min_day min(eval(if(vers=="5.0", _time, null()))) as min_day_50 dc(day) as days values(license) as license by guid | eval type = if(match(vers,"4.*"), "upgrade", "not upgrade") + "/" + if(days > 1, "repeat", "not repeat")| search license=enterprise | eval _time = min_day_50| timechart count by type| streamstats sum(*) as *
- 16. Data Model Goals •Makeiteasytoshare/reusedomainknowledge • Admins/powerusersbuilddatamodels • Non-technicalusersinteractwithdataviapivotUI
- 17.
- 18. What is aData Model? Adatamodelisasearch-timemappingofdataontoahierarchicalstructure Encapsulatetheknowledgeneededtobuildasearch Pivotreportsarebuildontopofdatamodels Data-independent Screenshot here
- 19. A Data Modelis a Collection of Objects Screenshot here
- 20. Objects Have Constraintsand Attributes Screenshot here
- 21. Child Objects InheritConstraints and Attributes Screenshot here
- 22. Child Objects InheritConstraints and Attributes
- 23.
- 24. Three Root ObjectTypes Event –MapstoSplunkevents – Requiresconstraints andattributes
- 25. Three Root ObjectTypes Event –MapstoSplunkevents – Requiresconstraints andattributes Search – MapstoarbitrarySplunksearch(may includegenerating,transformingand reportingsearchcommands) – Requiressearchstringattributes • Transaction – MapstogroupsofSplunkeventsor groupsofSplunksearchresults – Requiresobjectstogroup,fields/ conditionstogroupby,andattributes
- 26. Three Root ObjectTypes Event –MapstoSplunkevents – Requiresconstraints andattributes Search – MapstoarbitrarySplunksearch(may includegenerating,transformingand reportingsearchcommands) Requiressearchstringattributes Transaction – MapstogroupsofSplunkeventsor groupsofSplunksearchresults – Requiresobjectstogroup,fields/ conditionstogroupby,andattributes
- 27.
- 28.
- 29.
- 30. Best Practices Use eventobjects as often as possible – Benefit from data model acceleration Resist the urge to use search objects instead of event objects!! – Event based searches can be optimized better Minimize object hierarchy depth when possible – Constraint based filtering is less efficientdeeper down the tree Event object with deepest tree (and most matching results) first – Model-wide acceleration only for first event object and its descendants
- 31.
- 32. From Data Modelsto Reports
- 33. Usingthe UI Subhead Count ofhttp_success events, split by useragent events fields
- 34. Under the Hood:Object SearchString Generation Syntax: <constraints search> | <my attribute definitions> Example: sourcetype=access_* OR sourcetype=iis* uri=* uri_path=* status=* clientip=* referer=* useragent=*
- 35. Under the Hood:Object SearchString Generation Syntax: <base search> | <my attribute definitions> Example: _time=* host=* source=* sourcetype=* uri=* status<600 clientip=* referer=* useragent=* (sourcetype=access_* OR source=*.log) | eval userid=clientip | stats first(_time) as earliest, last(_time) as latest, list(uri_path) as uri_list by userid | earliest=* latest=* uri_list=*
- 36. Under the Hood:Object SearchString Generation Syntax: <objects to group search> | transaction <group by fields> <group by params> | <my attribute definitions> Example: sourcetype=access_* uri=* uri_path=* status=* clientip=* referer=* useragent=* | transaction clientip useragent | eval landingpage=mvindex(uri_path,1) | eval exitpage=mvindex(uri_path,-1)
- 37. Under the Hood:Object SearchString Generation Syntax: <parent object search> | search <my constraints> | <my attribute definitions> Example: sourcetype=access_* uri=* uri_path=* status=* clientip=* referer=* useragent=* status=2* | <my attribute definitions>
- 38. Usingthe SplunkSearch Language |datamodel <modelname> <objectID> search Example: | datamodel WebIntelligence HTTP_Request search Behindthescenes: sourcetype=access_* OR sourcetype=iis* uri=* uri_path=* status=* clientip=* referer=* useragent=*
- 39. Under the hood:Pivot Search String Generation Example: (sourcetype=access_* OR sourcetype=iis*) status=2* uri=* uri_path=* status=* clientip=* referer=* useragent=* | stats count AS "Count of HTTP_Sucess" by ”useragent" | sort limit=0 "useragent" | fields - _span | fields "useragent" "Count of HTTP_Success" | fillnull "Count of HTTP_Success" | fields "useragent" *
- 40. Usingthe SplunkSearch Language |pivot <modelname> <objectID> [statsfns, rowsplit, colsplit, filters, …] Example: | pivot WebIntelligence HTTP_Request count(HTTP_Request) AS "Count of HTTP_Request" SPLITROW status AS "status" SORT 0 status Behindthescenes: sourcetype=access_* OR sourcetype=iis* uri=* uri_path=* status=* clientip=* referer=* useragent=* | stats count AS "Count of HTTP_Request" by "status" | sort limit=0 "status" | fields - _span | fields "status", "Count of HTTP_Request" | fillnull "Count of HTTP_Request" | fields "status" *
- 41. Warnings • | datamodeland| pivot aregeneratingcommands • Theymustbeatthebeginningofthesearchstring • UseobjectIDsNOTuser-visibleobjectnames
- 42. Demo Building a reportfrom a data model
- 43.
- 44. Data Model onDisk EachdatamodelisaseparateJSONfile Livesin<myapp>/local/data/models (or<myapp>/default/data/modelsforpre- installedmodels) Hasassociatedconfstanzas andmetadata
- 45. Editing Data ModelJSON Atyourownrisk! ModelseditedviatheUIarevalidated Manuallyediteddatamodels:NOTSUPPORTED Exception:installinganewmodelbyaddingthefileto<myapp>/<local ORdefault>/data/modelsisprobablyokay
- 46. Deleting a DataModel UsetheUIforappropriatecleanup Potentialforbadstateifmanuallydeletingmodelondisk
- 47. InteractingWith a DataModel UsedatamodelbuilderandpivotUI–safestoption! UseRESTAPI–fordevelopers(seedocsfordetails) Use|datamodeland|pivotSplunksearchcommands
- 48.
- 49. Data Model Acceleration Runa pivot report Poll: are there new accelerated models? Turn on acceleration via UI Setting written to conf file Kick off collection Acceleration Kick off ad-hoc acceleration and run search Run search using on-disk acceleration Admin or power user Backend magic Non-technical user No acceleration
- 50. Model-WideAcceleration Pivotsearch: | tstats countAS "Count of HTTP_Success" from datamodel="WebIntelligence" where (nodename="HTTP_Request") (nodename="HTTP_Request.HTTP_Success") prestats=true | stats count AS "Count of HTTP_Success” Only accelerates first event- based object and descendants Does not accelerate search and transaction-based objects
- 51. Ad-Hoc Object Acceleration Kickoffaccelerationonpivotpage(re)loadfornon-acceleratedmodelsand search/transactionobjects Amortizecostofad-hocaccelerationoverrepeatedpivotingon sameobject Pivotsearch: |tstats count AS "Count of HTTP_Success" from sid=1379116434.663 prestats=true | stats count AS "Count of HTTP_Success”
- 52.
- 53.
Editor's Notes
- #2 Welcome to SplunkLive [City]. Thank you for taking the time to attend today’s event.
- #4 How can you leverage Splunk?
- #5 Splunk 6 takes large-scale machine data analytics to the next level by introducing three breakthrough innovations: Pivot – opens up the power of Splunk search to non-technical users with an easy-to-use drag and drop interface to explore, manipulate and visualize data Data Model – defines meaningful relationships in underlying machine data and making the data more useful to broader base of non-technical users Analytics Store – patent pending technology that accelerates data models by delivering extremely high performance data retrieval for analytical operations, up to 1000x faster than Splunk 5 Let’s dig into each of these new features in more detail.
- #6 How does the Analytics Store, Data Model and Pivot benefit users across the enterprise? Lets start with the IT Professional – this includes the Splunk Administrator or an advanced Splunk user that is familiar with SPL. Using Splunk 6 they can: Create data models Share data models with other users – delivering a consistent view of the data Accelerate data models using the Analytics Store Create reports using Pivot (although being power users, they may prefer using SPL directly!) Next we have the enterprise developer. Using Splunk 6 they can: Leverage data models built by IT, making searches more portable (using common Data Models ensures predictability of results) Leverage the Pivot interface in custom enterprise apps Finally, there are additional users that can now benefit – for example, the business or data analyst. Using Splunk 6 they can: Create reports, dashboards, charts and other visualizations using the Pivot interface and based on data models that provide an abstracted view of the raw data. Splunk 6 is not meant to replace existing BI and Business Analytics tools, but it does provide new visibility, insights and intelligence from operational data that can be used by business analysts to augment these tools. Data from Splunk software can also be leveraged directly using the Splunk API and SDKs and integrated into existing business analytics tools. For example, the recently announced Pentaho Business Analytics for Splunk® Enterprise (http://apps.splunk.com/app/1554), enables business users to utilize Pentaho to rapidly visualize and gain additional insights from Splunk’s machine data platform using existing in-house skills.
- #8 How can you leverage Splunk?
- #9 How can you leverage Splunk?
- #14 How can you leverage Splunk?
- #15 - The Splunk search language is very expressive. - Can perform a wide variety of tasks ranging from filtering to data munging and reporting - There are various search commands for complex transformations and statistics (e.g. correlation, prediction etc)
- #16 What does the search do? Basically, first it normalizes the individual accesses, which should be representable as a model object. Next it aggregates by guid to create an "instance" object, which should be representable in a DM. It calculates a field on that instance object, "type". Then it builds a timechart. of those, using a special "_time" value. Low overhead to start but learning curve quickly gets steep Obtaining website usage metrics should not require understanding Apache vs IIS format Admins won’t know apriori what questions are being asked of the data…so they can’t provide canned dashboards for all scenarios Backup search for example: eventtype=pageview | eval stage_2=if(searchmatch("uri=/download*"), _time, null()) | eval stage_1=if(searchmatch("uri=/product*"), _time, null()) | eval stage_3=if(searchmatch("uri=*download_track*"), _time, null()) | stats min(stage_*) as stage_* by cookie | search stage_1=* | where isnull(stage_2) OR stage_2 >= stage_1 | where isnull(stage_3) OR stage_3 >= stage_2 | eval stage = case(isnull(stage_2), "stage_1", isnull(stage_3), "stage_2", 1==1, "stage_3") | stats count by stage | reverse | accum count as cumulative_count | reverse | streamstats current=f max(cumulative_count) as stage_1_count last(cumulative_count) as prev_count
- #20 What are the important “things” in your data? E.g. WebIntelligence might have HTTPAccess HTTPSuccess User Session How are they related? There’s more than one “right” way to define your objects
- #21 Constraints filter down to a set of a data Attributes are the fields and knowledge associated with the object Both are inherited!
- #22 A child object is a type of its parent object: e.g. An HTTP_Success object is a type of HTTP_Access Adding a child object is essentially a way of adding a filter on the parents A parent-child relationship makes it easy to do queries like “What percentage of my HTTP_Access events are HTTP_Success events?”
- #24 How can you leverage Splunk?
- #25 Constraints are essentially the search broken down into a hierarchy, attributes are the associated fields and knowledge
- #26 Arbitrary searches that include transforming commands to define the dataset that they represent Fix example here? TODO
- #27 Enable the creation of objects that represent transactions Use fields that have already been added to the model via event or search objects
- #28 This is how we capture knowledge
- #29 Required: Only events that contain this field will be returned in Pivot Optional: The field doesn't have to appear in every event Hidden: The field will not be displayed to Pivot users when they select the object in Pivot Use this for fields that are only being used to define another attribute, such as an eval expression Hidden & Required: Only events that contain this field will be returned, and the field will be hidden from use in Pivot
- #31 Be careful about lookup permissions – must be available in the context where you want to use them
- #33 How can you leverage Splunk?
- #44 How can you leverage Splunk?
- #50 Divanny will help make this slide MUCH prettier
- #51 This could be more slides, more details
- #53 This could be more slides, more details
- #54 How can you leverage Splunk?