![Extract Fields
25
Interactive Field Extractor
• generate PCRE
• editable regex
• preview/save
props.conf
[mysourcetype]
REPORT-myclass = myFields
transforms.conf
[myFields]
REGEX = ^(w+)s
FORMAT = myFieldLabel::$1
Configuration File
• manual field extraction
• delim-based extractions
Rex Search Command
... | rex field=_raw "From: (?<from>.*) To:
(?<to>.*)"](https://image.slidesharecdn.com/2014-04-08gettingstartedwithsplunkbreakoutsessionzurichpublic-140404064423-phpapp01/75/SplunkLive-Zurich-2014-Beginner-Workshop-Getting-started-with-Splunk-25-2048.jpg)
Uploaded byGeorg Knon
SplunkLive! Zürich 2014 Beginner Workshop: Getting started with Splunk
The document is an agenda for a Splunk technical workshop on getting started with Splunk user training. The agenda covers installing and starting Splunk, performing searches, creating alerts and dashboards, deployment and integration functionality, and getting support through the Splunk community.
More Related Content
Similar to SplunkLive! Zürich 2014 Beginner Workshop: Getting started with Splunk
SplunkLive! Zürich 2014 Beginner Workshop: Getting started with Splunk
- 1. Copyright © 2013Splunk Inc. Zürich, 8. April , 2014 Technical Workshops Getting Started User Training Getting Started User Training Workshop Holger Sesterhenn Senior Sales Engineer
- 2. Agenda • Getting Startedwith Splunk • Search • Alert • Dashboard • Deployment and Integration • Community • Help & Questions 2
- 3. Getting Started WithSplunk
- 4. IT Operations Security and Compliance Digital Intelligence App Dev and AppMgmt. Developer Platform (REST API, SDKs) Business Analytics Industrial Data and Internet of Things Small Data. Big Data. Huge Data. Splunk Delivers Value Across IT and the Business
- 5. Install Splunk Splunk Home •WIN: Program FilesSplunk • Other: /opt/splunk (Applications/splunk) Start Splunk • WIN: Program FilesSplunkbinsplunk.exe start (services start) • *NIX: /opt/splunk/bin/splunk start www.splunk.com/download • 32 or 64 Bit? • Indexer or Universal Forwarder?
- 6. Splunk Licenses Free DownloadLimits Indexing to 500MB/day • Enterprise Trial License expires after 60 days • Reverts to Free License Features Disabled in Free License • Multiple user accounts and role-based access controls • Distributed search • Forwarding to non-Splunk Instances • Deployment management • Scheduled saved searches and alerting • Summary indexing Other License Types • Enterprise, Forwarder, Trial
- 7. Default installation on:http://localhost:8000 7 Splunk Web Basics Browser Support • Firefox 10.x and latest • Internet Explorer 7, 8, 9 and 10 • Safari (latest) • Chrome (latest) Index data • Add data • Getting Started App • Install an App (Splunk for Windows, *NIX)
- 8. 8 Splunk Web Basicscontinued… Splunk Home • Provides Interactive portal to the Apps & data. • Includes a search bar and three panels: 1 – Apps 2 – Data 3 - Help Splunk Apps • Splunk Home Find more apps • Provide different contexts for your data out of sets of views, dashboards, and configurations • Default Search App • You can create your own!
- 9. Optional: add sometest data Download the sample file, follow this link and save the file to your desktop, then unzip: http://www.splunkbook.com (Using Splunk Book) To add the file to Splunk: – From the Welcome screen, click Add Data. – Click From files and directories on the bottom half of the screen. – Select Skip preview. – Click the radio button next to Upload and index a file. – Click Save. 9
- 10. Best Practice Suggestion: Createan individual Index based on sourcetype. • Easier to re-index data if you make a mistake. • Easier to remove data. • Easier to define permissions and data retention. 10
- 11.
- 12. Search app –Summary viewcurrent view global stats app navigation time range picker Selecting Data Summary: • Host • Source • Sourcetype start search search box
- 13. Searching 13 Search > * SelectTime Range • Historical, custom, or real-time Select Mode • Smart, Fast, Verbose Using the timeline • Click events and zoom in and out • Click and drag over events for a specific range
- 14. 14 Everything is searchable Everythingis searchable • * wildcards supported • Search terms are case insensitive • Booleans AND, OR, NOT – Booleans must be uppercase – Implied AND between terms – Use () for complex searches • Quote phrases fail* fail* nfs error OR 404 error OR failed OR (sourcetype=access_*(500 OR 503)) "login failure"
- 15.
- 16. Search Assistant 16 Contextual Help -advanced type-ahead History - search - commands Search Reference - short/long description - examples suggests search terms updates as you type shows examples and help toggle off / on
- 17. Searches can bemanaged as asynchronous processes Jobs can be • Scheduled • Moved to background tasks • Paused, stopped, resumed, finalized • Managed • Archived • Cancelled Job Management Modify Job Settings pause finalize delete 17
- 18. Search Commands 18 Search >error | head 1 Search results are “piped” to the command Commands for: • Manipulating fields • Formatting • Handling results • Reporting
- 19. Over 130 Commands! 19 splunk.com> Documentation > Search Reference abstract accum addcoltotals addinfo addtotals af analyzefields anomalies anomalousvalue append appendcols ar associate audit autoregress bin bucket chart cluster collect common contingency convert correlate counttable crawl ctable dbinspect dedup delete delta diff discretize erex eval eventcount eventstats excerpt extract file fillnull folderize format gentimes head highlight iconify input inputcsv inputlookup iplocation join kmeans kv kvform loadjob localize localop lookup macro makecontinuous makemv maketable map metadata multikv mvcombine mvexpand nomv outlier outlierfilter outputcsv outputlookup outputtext overlap rangemap rare regex relevancy rename replace reverse run savedsearch savedsplunk script scrub selfjoin sendemail set sichart sirare sistats sitimechart sitop slc stash strcat streamstats sumindex summaryindex tail test timechart top transaction transam trendline typeahead typelearner typer uniq untable xmlkv xmlunescape xpath xyseries http://www.splunk.com/base/Documentation/latest/SearchReference/SearchCheatsheet
- 20.
- 21. Fields 21 Default fields • host,source, sourcetype, linecount, etc. • View on left panel in search results or all in field picker Where do fields come from? • Pre-defined by sourcetypes • Automatically extracted key-value pairs • User defined
- 22. Sources, Sourcetypes, Hosts •Host - hostname, IP address, or name of the network host from which the events originated • Source - the name of the file, stream, or other input • Sourcetype - a specific data type or data format 2 2
- 23. 23 Tagging and EventTyping Eventtypes for more human-readable reports • to categorize and make sense of mountains of data • punctuation helps find events with similar patterns Search > eventtype=failed_login instead of Search > “failed login” OR “FAILED LOGIN” OR “Authentication failure” OR “Failed to ………………authenticate user” Tags are labels • apply ad-hoc knowledge • create logical divisions or groups • tag hosts, sources, fields, even eventtypes Search > tag=web_servers instead of Search > host=“apache1.splunk.com” OR host=“apache2.splunk.com” OR …………….host=“apache3.splunk.com”
- 24. Extract Fields 24 Interactive FieldExtractor • generate PCRE • editable regex • preview/save
- 25. Extract Fields 25 Interactive FieldExtractor • generate PCRE • editable regex • preview/save props.conf [mysourcetype] REPORT-myclass = myFields transforms.conf [myFields] REGEX = ^(w+)s FORMAT = myFieldLabel::$1 Configuration File • manual field extraction • delim-based extractions Rex Search Command ... | rex field=_raw "From: (?<from>.*) To: (?<to>.*)"
- 26. Saved Search &Alert Basics
- 27. Saved Searches 27 Leverage Searchesfor future Insights! • Reports • Dashboards • Alerts • Eventtypes Add a Time Range Picker • Preset • Relative • Real-time • Date-Range • Date & Time Range • Advanced
- 28. Create Alerts 28 Scheduled orReal-Time • Define Time Ranges • Conditions • Thresholds
- 29. Alerting Continued… 29 Searches runon a schedule and fire an alert • Example: Run a search for “Failed password” every 15 min over the last 15 min and alert if the number of events is greater than 10 Searches are running in real-time and fire an alert • Example: Run a search for “Failed password user=john.doe” in a 1 minute window and alert if an event is found
- 30. Alerting Actions 30 • Sendemail • RSS • Execute a script • Track Alert Details
- 31. Report & DashboardWackiness
- 32. Reporting 32 results of anysearch Define your Search and set your time range, accelerate you search and more Choose the type of chart (line, area, column, etc) and other formatting options Build reports from
- 33. Reporting Examples 33 • Usewizard or reporting commands (timechart, top, etc) • Build real-time reports with real-time searches • Save reports for use on dashboards
- 34. Dashboards 34 Create dashboards fromsearch results
- 35.
- 36. Manager Settings 36 For Allof that Cool Stuff You Just Created (and more!) • Permissions • Saved Searches/Reports • Custom Views • Distributed Splunk • Deployment Server • License Usage….
- 37.
- 38. Splunk Has FourPrimary Functions 38 • Searching and Reporting (Search Head) • Indexing and Search Services (Indexer) • Local and Distributed Management (Deployment Server) • Data Collection and Forwarding (Forwarder) A Splunk install can be one or all roles…
- 39. Getting Data IntoSplunk 39 Agent and Agent-less Approach for Flexibility perf shell code Mounted File Systems hostnamemount syslog TCP/UDP WMI Event Logs Performance Active Directory syslog compatible hosts and network devices Unix, Linux and Windows hosts Windows hosts Custom apps and scripted API connections Local File Monitoring log files, config files dumps and trace files Windows Inputs Event Logs performance counters registry monitoring Active Directory monitoring virtual host Windows hosts Scripted Inputs shell scripts custom parsers batch loading Agent-less Data Input Splunk Forwarder
- 40. Understanding the UniversalForwarder 40 Forward data without negatively impacting production performance. Scripts Universal Forwarder Deployment Logs ConfigurationsMessages Metrics Central Deployment Management Monitor files, changes and the system registry; capture metrics and status. Universal Forwarder Regular (Heavy) Forwarder Monitor All Supported Inputs ✔ ✔ Routing, Filtering, Cloning ✔ ✔ Splunk Web ✔ Python Libraries ✔ Event Based Routing ✔ Scripted Inputs ✔
- 41. Horizontal Scaling 41 Load balancedsearch and indexing for massive, linear scale out. Forwarder Auto Load Balancing Distributed Search
- 42. Multiple Datacenters 42 Headquarters London HongKong Tokyo New York Distributed Search Index and store locally. Distribute searches to datacenters, networks & geographies.
- 43. High Availability, OnCommodity Servers and Storage 43 As Splunk collects data, it keeps multiple identical copies If indexer fails, incoming data continues to get indexed Indexed data continues to be searchable Easy setup and administration Data integrity and resilience without a SAN Index Replication Splunk Universal Forwarder Pool Constant Uptime
- 44. Service Desk Event Console SIEM SendData to Other Systems 44 Route raw data in real time or send alerts based on searches.
- 45. Integrate External Data 45 LDAP,AD Watch Lists CRM/ER P CMDB Correlate IP addresses with locations, accounts with regions Extend search with lookups to external data sources.
- 46. Integrate Users andRoles 46 Problem Investigation Problem Investigation Problem Investigation Save Searches Share Searches LDAP, AD Users and Groups Splunk Flexible Roles Manage Users Manage Indexes Capabilities& Filters NOT tag=PCI App=ERP … Map LDAP & AD groups to flexible Splunk roles. Define any search as a filter. Integrate authentication with LDAP and Active Directory.
- 47. Centralized Licensing Management 47 ProblemInvestigation Groups, Stacks, and Pools for Enterprise Deployments
- 48. Deployment Monitoring 48 Keep TabsOn Your Splunk Enterprise Deployment ForwardersIndexersSourcetypesLicenses
- 49.
- 50. Support Through theSplunk Community 50 Browse and share Apps from Splunk, Partners and the Community apps.splunk.com Splunkbase Community-driven knowledge exchange and Q&A answers.splunk.com 5 tracks, more than 40 sessions, the smartest Splunk users together conf.splunk.com .conf2014
- 51. Where to Gofor Help 51 • Documentation – http://www.splunk.com/base/Documentation • Technical Support – http://www.splunk.com/support • Videos – http://www.splunk.com/videos • Education – http://www.splunk.com/goto/education • Community – http://answers.splunk.com – http://apps.splunk.com • Splunk Book – http://splunkbook.com
- 52. Thank you November 12st, 2012 Technical Workshops GettingStarted User Training