Data Onboarding Breakout Session
•
0 likes•1,516 views
This document provides an overview and examples of data onboarding in Splunk. It discusses best practices for indexing data, such as setting the event boundary, date, timestamp, sourcetype and source fields. Examples are given for onboarding complex JSON, simple JSON and complex CSV data. Lessons learned from each example highlight issues like properly configuring settings for nested or multiple timestamp fields. The presentation also introduces Splunk capabilities for collecting machine data beyond logs, such as the HTTP Event Collector, Splunk MINT and the Splunk App for Stream.
More Related Content
What's hot
What's hot (20)
Similar to Data Onboarding Breakout Session
Similar to Data Onboarding Breakout Session (20)
More from Splunk
More from Splunk (20)
Recently uploaded
Recently uploaded (20)
Data Onboarding Breakout Session
- 1. Copyright © 2015 Splunk Inc. Data Onboarding 101 KirkHanson SalesEngineering
- 2. Agenda 1 Splunk Platform – a refresher Data Onboarding – live looks TA’s & Apps what are they? Go Beyond The Logs Q&A 2 3 4 5
- 3. 3 Make machine data accessible, usable and valuable to everyone. 3
- 4. Big Data Comes from Machines Volume | Velocity | Variety | Variability 4 GPS RFID Hypervisor Web Servers Email Messaging Clickstreams Mobile Telephony IVR Databases Sensor Servers Telematics Storage Security Devices Desktops
- 5. 6 Machine Data Sources Order Processing Twitter Care IVR Middleware Error
- 6. 7 Machine Data Contains Critical Insights Order ID Customer’s Tweet Time Waiting On Hold Product ID Company’s Twitter ID Order ID Customer ID Twitter ID Customer ID Customer ID Sources Order Processing Twitter Care IVR Middleware Error
- 7. 88 Turn Machine Data into Operational Intelligence INDEX ANY MACHINE DATA: ANY SOURCE, TYPE, VOLUME GAIN REAL-TIME VISIBILITY Online Services Web Services Servers Security GPS Location Storage Desktops Networks Packaged Applications Custom ApplicationsMessaging Telecoms Online Shopping Cart Web Clickstreams Databases Energy Meters Call Detail Records Smartphones and Devices RFID On- Premises Private Cloud Public Cloud Application Delivery Security and Compliance Infrastructure Monitoring Business Analytics Internet of Things 8
- 9. Six things to get right at Index time Event Boundary / LineBreaking Date Timestamp Sourcetype Source Host Index
- 10. Best Practices Local before Prod Confirm Sourcetype Separate Index Specific as possible Try before you buy Save not override
- 11. Data Onboarding
- 12. 1414 Data Onboarding Examples Complex JSON Configured Settings Simple JSON Default Settings Complex CSV Minimal Settings
- 13. 1515 Data Onboarding Examples Complex JSON Configured Settings Simple JSON Default Settings Complex CSV Minimal Settings
- 14. Data Onboarding Live Look (simple JSON)
- 15. 1717 Simple JSON – Lessons Learned Complex JSON Configured Settings Simple JSON Default Settings Complex CSV Minimal Settings • Structured • TimeStamp found in first event • Smaller set of data
- 16. 1818 Complex CSV onboarding Complex JSON Configured Settings Complex CSV Minimal Settings Simple JSON Default Settings • Structured • TimeStamp found in first event • Smaller set of data
- 17. Data Onboarding Live Look (CSV)
- 18. 2020 Complex CSV - Lessons Learned Complex JSON Configured Settings Complex CSV Minimal Settings Simple JSON Default Settings • Structured • TimeStamp found in first event • Smaller set of data • TimeStamp not found • Data otherwise standard
- 19. 2121 Complex JSON Onboarding Complex JSON Configured Settings Simple JSON Default Settings • Structured • TimeStamp found in first event • Smaller set of data Complex CSV Minimal Settings • TimeStamp not found • Data otherwise standard
- 20. Data Onboarding Live Look (complex JSON)
- 21. 2323 Complex JSON - Lessons Learned Complex JSON Configured Settings Simple JSON Default Settings • Structured • TimeStamp found in first event • Smaller set of data Complex CSV Minimal Settings • TimeStamp not found • Data otherwise standard •Nested •Multiple TimeStamp fields •Larger single event
- 22. Why reinvent the wheel Splunkbase (apps & TA’s)
- 23. A Growing, Global Community of Users Dev.splunk.com40,000+ questions and answers 1000+ apps Local User Groups and SplunkLive! events
- 24. Web Framework SDKs REST API Log directly to Splunk Extract Splunk data for archiving Integrate with third-party reporting tools and portals Integrate Splunk search results into your application . . . and more The Splunk Platform User&DeveloperInterfaces Ticketing/Help Desk Custom Biz Applications Inputs,Apps,OtherContent Scripted inputs (.sh, .py, .bat, .ps1, etc.) Get data from APIs and other remote data interfaces and message queues. Databases (JDBC) Splunk DB Connect lets you enrich and combine your machine data with database data. Network events (TCP, UDP, SNMP, NetFlow, HTTP(S)) Get data from any network port, SNMP events, or send your application data directly via HTTP (or HTTPS) through HTTP Event Collector. Forwarders (TCP) Gather machine and historical data (e.g. text-based files, Windows event logs, Active Directory). Modular inputs (Stream data as plain text or XML) Extend the Splunk Enterprise framework to define a custom input capability (e.g. Twitter, S3, Splunk MINT). Enrich and extend the usefulness of your event data through interactions with external resources like asset info, employee info, threat feeds, honeypots, and more. External lookups (.py or .csv) Customize the Splunk Web UI Real time data collection, indexing and search, as well as alerting, large scale distributed processing, user authentication (through Splunk’s built-in system, LDAP or a scripted authentication API for use with an external authentication system), and role-based access control. CoreEngine Business Intelligence ODB C Capture wire data from endpoints and key network locations with the Splunk App for Stream.Wire data Systems Management Infrastructure Apps XenApp XenDesktop Cloud Services Examples Mainframe Other Monitoring Splunk Premium Solutions Server, Storage, Network Examples
- 25. App? ?? Add-on??
- 26. Splunkbase by the #’s IOT (49) Application Management (158) IT Ops (381) Security & Compliance (384) Business Analytics (70) Utilities 292 Cool Stuff 210
- 27. Go Beyond the Logs
- 28. 3030 Which input(s) do we NOT support today? 1) Text-based files 2) Windows sources 3) TCP / UDP ports 4) SMNP events 5) NetFlow 6) HTTP(S) 7) FIFO queues 8) Scripted inputs 9) Message queues 10)Modular inputs 11)Databases 12)External lookups 13)Wire data 14)SDK
- 29. 3131 We support ALL of these inputs! 31
- 30. 3232 Our focus today 1) Text-based files 2) Windows sources 3) TCP / UDP ports 4) SMNP events 5) NetFlow 6) HTTP(S) 7) FIFO queues 8) Scripted inputs 9) Message queues 10) Modular inputs (specifically MINT) 11) Databases 12) External lookups 13) Wire data 14) SDK
- 32. 3434 Http Event Collect 34 EC HTTP or HTTPS POST <protocol>://<host>:<mPort>/services/collector(/raw) Indexer Search Head Event Source(s)
- 33. Splunk Mint Data Collector
- 34. 3636 Splunk> Mobile Intelligence 36 MINT SDKs MINT Data Collector MINT App MINT Management Console
- 35. Splunk App for Stream
- 36. 3838 Stream Concept Users Search Head(s)* Indexer(s) Universal Forwarder + TA Physical Data Center Physical or Virtual ServersEnd Users Internet Firewall Public or Private Cloud LOCAL COLLECTION + TA + splunk_app_stream
- 38. Six things to get right at Index time Event Boundary / LineBreaking Date Timestamp Sourcetype Source Host Index
- 39. Learn More from Today
- 40. 4343 Where to go to learn more Data Pipeline – http://goo.gl/FP3JTM Distributed Deployment Manual – http://goo.gl/MTJr0K How Indexing works (the data pipeline) – https://goo.gl/SGRC1y Tutorial & tutorial data – http://goo.gl/OYNCnc Date and time format variables – http://goo.gl/E9Onpq 43
- 41. 4444 Resources: HTTP Event Collector • Introduction to Splunk HTTP Event Collector (Developer Portal) • Set up and use HTTP Event Collector (Docs) • Troubleshooting HTTP Event Collector (Confluence) • HTTP Event Collector, your DIRECT event pipe to Splunk 6.3 (Blogs: Tips & Tricks) • Liberate Your Application Logging (.conf2015) 44 EC
- 42. 4545 Resources: MINT • Splunk MINT Manual (Docs) • Start with Splunk MINT SDKs (Management Console) • Getting Started with Splunk MINT (Blogs: Mobile) • Splunk MINT: Security & Privacy (Blogs: Mobile) • What's the difference between MINT Management Console and the Splunk MINT App? 45
- 43. 4646 Resources: Stream • Performance test results and recommendations (Docs) • Supported protocols (Docs) • Splunk App for Stream 6.4 (TEC) • Everything you always wanted to know about SPAN ports, Network Taps, Packet Mirrors, and the Splunk App for Stream (but were afraid to ask) (Blogs: Security) • How Can You Use Ephemeral Streams? (Blogs: Tips & Tricks) 46
- 44. 47 Northern Cal Tech Talks! Monthly WebEx Sessions • Ted Talk style presentation • Q&A Chat forum So what’s next on the agenda? • March 23rd @ 10AM PST - Building & Deploying Apps. • April 20th @ 10AM PST - Top 5 most useful search commands. See more at: http://live.splunk.com/NorCalTechTalks
- 45. 48 SEPT 26-29, 2016 WALT DISNEY WORLD, ORLANDO SWAN AND DOLPHIN RESORTS • 5000+ IT & Business Professionals • 3 days of technical content • 165+ sessions • 80+ Customer Speakers • 35+ Apps in Splunk Apps Showcase • 75+ Technology Partners • 1:1 networking: Ask The Experts and Security Experts, Birds of a Feather and Chalk Talks • NEW hands-on labs! • Expanded show floor, Dashboards Control Room & Clinic, and MORE! The 7th Annual Splunk Worldwide Users’ Conference PLUS Splunk University • Three days: Sept 24-26, 2016 • Get Splunk Certified for FREE! • Get CPE credits for CISSP, CAP, SSCP • Save thousands on Splunk education!
- 46. Q&A
- 47. Thank You!!
- 48. 5353 Complex JSON Configured settings •Nested •Multiple TimeStamp fields •Larger single event Simple JSON Default settings •Structured •TimeStamp found in first event •Smaller set of data Complex JSON – Lessons Learned
- 49. Scalable Syslog Event Collection Dedicated Syslog Collector with Splunk Forwarder Splunk Forwarder With Syslog Listener
Editor's Notes
- At Splunk, our mission is to make machine data accessible, usable and valuable to everyone. And this overarching mission is what drives our company and product priorities.
- What is this machine data, and why is it a big deal? Well, it’s one of the fastest growing, most complex and most valuable segments of data. All the webservers, applications, network devices, mobile devices, sensors – all of the technology infrastructure running your enterprise – generates massive streams of data, in an array of unpredictable formats that are difficult to process and analyze by traditional methods or in a timely manner. Why is this “machine data” valuable? Because it contains a trace - a categorical record - of user behavior, cyber-security risks, application behavior, service levels, fraudulent activity and customer experience. Characteristics of machine data – the four V’s - the last two are the most interesting / challenging.
- All the webservers, applications, network devices – all of the technology infrastructure running an enterprise or organization – generates massive streams of data, in an array of unpredictable formats that are difficult to process and analyze by traditional methods or in a timely manner.
- Let’s take a closer look at machine data
- To frame our discussion, let’s use this example of purchasing a product from your tablet or smartphone: the purchase transaction fails, you call the call center and then tweet about your experience. All these events are captured - as they occur - in the machine data. Each of the underlying systems hast the potential to generate millions of machine data events daily. Here we see small excerpts from just some of them. When we look more closely at the data we see that it contains valuable information – right down to what was tweeted. What’s important, is first of all, the ability to actually see across all these data sources, but then also to correlate related events and provide meaningful insight. If you can correlate and visualize the data, you can build a picture of activity, behavior and experience. And what if you can do all of this in real-time? You can respond more quickly to events that matter. This example ties into your scenario but you can also extrapolate this example to a wide range of use cases – security and fraud, transaction monitoring and analysis, web analytics, IT operations and so on.
- Our customers typically start with Splunk to solve a specific problem, and then expand from there to address a broad range of use cases, across application troubleshooting, IT infrastructure monitoring, security, business analytics, Internet of things, and many others that are entirely innovated by our customers. Here’s how it works. Splunk software and cloud services reliably collect and index machine data, from a single source to tens of thousands of sources. All in real time. - Once data is in Splunk, you can search, analyze, report-on and derive business value from all your data
- Since 2004 Splunk has become the platform of choice to extract business value from Machine data. That means that Splunk collects, indexes, analyzes, reports and predicts on machine-generated data from a single product. It’s an open platform with over 500 Splunk Apps available and allows for custom development.
- Setting a certain amount of meta data at ingestion that way its easier to extract value from it
- Input on local before Prod Confirm Sourcetype Separate Index When monitoring – be as specific as possible Try it before you buy it Save-as not override _____ Systematic way to bring new data sources into Splunk Enure that new data is instantly usable & has maximum value for users Goes hand-in-hand with the User Onboarding process Lookout for inadvertent, runaway monitor clauses Don’t monitor thousands of files unnecessarily– that’s the NSA’s job
- Introduce the idea thaw e are going to use the GUI For all Minimal effort overall
- DO WE DEFINE JSON?
- Went through this morning session using the tutorial data – now lets do something that is more structured
- Smaller events Point out that we can ingest csv and json – out of the box But what if ?? What if there is a csv with an out of norm
- Complex CSV %d%b%Y:%H:%M:%S.%2N %d Day of the month as a decimal number, includes a leading zero. (01 to 31) %b Abbreviated month name. (Jan, Feb, etc.) %Y Year as a decimal number with century. (2015) %H Hour (24-hour clock) as a decimal number. Hours represented by the values 00 to 23. Leading zeros are accepted but not required. %M %MMinute as a decimal number. Minutes represented by the values 00 to 59. %S %SSecond as a decimal number, for example 00 to 60. %N %NSubseconds with width. (%3N = milliseconds, %6N = microseconds, %9N = nanoseconds) http://docs.splunk.com/Documentation/Splunk/6.3.3/SearchReference/Commontimeformatvariables
- Why JSON? Still no header – this isn’t CSV – both a blessing and a curse
- Why JSON? Still no header – this isn’t CSV – both a blessing and a curse
- Why JSON? Still no header – this isn’t CSV – both a blessing and a curse
- With thousands of enterprise customers and an order of magnitude more actual users, we have a thriving community. We launched a dev portal a few months back and already have over 3,000 unique visitors per week. We have over 500 apps contributed by ourselves, our partners and our community. Our knowledge exchange Answers site has over 40,000+ questions answered. We host our annual users’ conference and local Splunk Live! Events where you can meet other users. Best of all, Splunk’s passionate and vibrant community demands more from Splunk and gives us incredible feedback, which drives us to constantly innovate and respond to their needs.
- Focus on the fact that there is a lot of sources of data and our goal any data anytime anywhere(any source)!
- Splunk apps and add-ons: what & why? Splunk apps allow developers to extend data ingestion and processing capabilities of Splunk Enterprise for your specific needs. Apps facilitate more efficient completion of domain-specific tasks by the end user. High-level perspective A Splunk app is a prebuilt collection of additional capabilities packaged for a specific technology, or use cases, which allows a more effective usage of Splunk Enterprise. You can use Splunk apps to gain the specific insights you need from your machine data. Depending on the type and complexity of those use cases, and also whether the developer wants certain app parts to be configured or distributed separately (potentially by a third party), an app may rely on various add-ons. An add-on is a technical component that can be re-used across a number of different use cases and packaged with one or more Splunk apps. Add-ons may contain one or more knowledge objects, which encapsulate a specific functionality focused on a single concern and its configuration. Using an add-on should help to reduce the technical risk and cost of building an app.
- Splunk apps and add-ons: what & why? Splunk apps allow developers to extend data ingestion and processing capabilities of Splunk Enterprise for your specific needs. Apps facilitate more efficient completion of domain-specific tasks by the end user. High-level perspective A Splunk app is a prebuilt collection of additional capabilities packaged for a specific technology, or use cases, which allows a more effective usage of Splunk Enterprise. You can use Splunk apps to gain the specific insights you need from your machine data. Depending on the type and complexity of those use cases, and also whether the developer wants certain app parts to be configured or distributed separately (potentially by a third party), an app may rely on various add-ons. An add-on is a technical component that can be re-used across a number of different use cases and packaged with one or more Splunk apps. Add-ons may contain one or more knowledge objects, which encapsulate a specific functionality focused on a single concern and its configuration. Using an add-on should help to reduce the technical risk and cost of building an app.
- I would like to start by taking a survey from this lovely audience on which inputs do we NOT support today?
- However, our focus today will be on the HTTP Event Collector, the MINT Data Collector and Splunk App for Stream.
- You start by sending events directly from sources like a server, docker, mobile device, IoT, or browser as raw JSON, loose text or XML <CLICK> across an HTTP or HTTPS POST request to our services/collector or services/collector/raw REST API endpoint. <CLICK>
- So, you’re at another lunch because we all need to eat. This time you came prepared with your own pen. You grab a napkin and start doodling how it works. It might look something like this: <CLICK> The Splunk MINT SDKs integrate into the iOS and/or Android mobile apps to collect data from those apps, and then <CLICK> send that data to the MINT Data Collector, which is a cloud service that we provide. The MINT Data Collector then <CLICK> forwards the data to Splunk MINT Management Console and the Splunk MINT App. That’s it!
- So, how does it work? Let’s say you are having lunch with Mr. Customer and you start chatting about the Splunk App for Stream. You ask the closest wait staff for a pen and grab a clean napkin. You then say to Mr. Customer, “let me doodle how it works”. There are three types of network collection architectures: (1) local, (2) SPAN or port mirror, and (3) TAP. Don’t fret if you cannot spell SPAN or TAP, I have included resources at the end of this presentation for you to check out on your own time. So, the doodle for local collection might look like this: <CLICK> First, the basic data flow from an end user into the customer’s environment. <CLICK> Then, in their Splunkland, they may have forwarders on their hosts sending data to their indexers. <CLICK> Finally, for local collection of their wire data, it requires the installation of Splunk_TA_stream on the forwarders of each host on the network or network segment that they want to monitor. The TA also needs to be installed on the indexers, and the app on the search heads. Please note that SHC is not supported at this time with Stream.
- Setting a certain amount of meta data at ingestion that way its easier to extract value from it
- In addition to live, .conf, docs, answers, meetups etc etc
- We’re headed to the East Coast! 2 inspired Keynotes – General Session and Security Keynote + Super Sessions with Splunk Leadership in Cloud, IT Ops, Security and Business Analytics! 165+ Breakout sessions addressing all areas and levels of Operational Intelligence – IT, Business Analytics, Mobile, Cloud, IoT, Security…and MORE! 30+ hours of invaluable networking time with industry thought leaders, technologists, and other Splunk Ninjas and Champions waiting to share their business wins with you! Join the 50%+ of Fortune 100 companies who attended .conf2015 to get hands on with Splunk. You’ll be surrounded by thousands of other like-minded individuals who are ready to share exciting and cutting edge use cases and best practices. You can also deep dive on all things Splunk products together with your favorite Splunkers. Head back to your company with both practical and inspired new uses for Splunk, ready to unlock the unimaginable power of your data! Arrive in Orlando a Splunk user, leave Orlando a Splunk Ninja! REGISTRATION OPENS IN MARCH 2016 – STAY TUNED FOR NEWS ON OUR BEST REGISTRATION RATES – COMING SOON!
- To prevent you walking out of this presentation feeling like this we thought it best to have a fake real-world scenario wherein we discussed a common question that comes up when talking about
- Normalizes data from different sources – Host and hostname discussion