SplunkLive! Munich 2018: Get More From Your Machine Data Splunk & AISplunk
Presented at SplunkLive! Munich 2018:
- Why AI & Machine Learning?
- What is Machine Learning?
- Splunk's Machine Learning Tour
- Use Cases & Customer Stories
SplunkLive! Munich 2018: Predictive, Proactive, and Collaborative ML with IT ...Splunk
Presented at SplunkLive! Munich 2018:
- What data do we need?
- We need Machine Learning
- Real Use Case Example
- Let's Drive Into How it Works
- Next Steps
SplunkLive! Munich 2018: Get More From Your Machine Data Splunk & AISplunk
Presented at SplunkLive! Munich 2018:
- Why AI & Machine Learning?
- What is Machine Learning?
- Splunk's Machine Learning Tour
- Use Cases & Customer Stories
SplunkLive! Munich 2018: Predictive, Proactive, and Collaborative ML with IT ...Splunk
Presented at SplunkLive! Munich 2018:
- What data do we need?
- We need Machine Learning
- Real Use Case Example
- Let's Drive Into How it Works
- Next Steps
SplunkLive! Frankfurt 2018 - Legacy SIEM to Splunk, How to Conquer Migration ...Splunk
Presented at SplunkLive! Frankfurt 2018:
Introduction
SIEM Migration Methodology
Use Cases
Datasources & Data Onboarding
ES Architecture
Third-Party Integrations
You Got This!
SplunkLive! Frankfurt 2018 - Get More From Your Machine Data with Splunk AISplunk
Presented at SpluknLive! Frankfurt 2018:
Why AI & Machine Learning?
What is Machine Learning?
Splunk's Machine Learning Tour
Use Cases & Customer Stories
Wrap Up
SplunkLive! Frankfurt 2018 - Data Onboarding OverviewSplunk
Presented at SplunkLive! Frankfurt 2018:
Splunk Data Collection Architecture
Apps and Technology Add-ons
Demos / Examples
Best Practices
Resources and Q&A
SplunkLive! Paris 2018: Use Splunk for Incident Response, Orchestration and A...Splunk
Presented at SplunkLive! Paris 2018:
- Challenges with Security Operations Today
- Overview of Splunk Adaptive Response Initiative
- Technology behind the Adaptive Response Framework
- Demonstrations
- How to build your own AR Action
- Resources
Splunk Discovery: Warsaw 2018 - Reimagining IT with Service IntelligenceSplunk
Presented at Splunk Discovery Warsaw 2018:
What's Service Intelligence and Why You Should Care
Introduction to Splunk IT Service Intelligence
IT Service Intelligence Key Concepts
Demo
Splunk Data Onboarding Overview - Splunk Data Collection ArchitectureSplunk
Splunk's Naman Joshi and Jon Harris presented the Splunk Data Onboarding overview at SplunkLive! Sydney. This presentation covers:
1. Splunk Data Collection Architecture 2. Apps and Technology Add-ons
3. Demos / Examples
4. Best Practices
5. Resources and Q&A
SplunkLive! Frankfurt 2018 - Legacy SIEM to Splunk, How to Conquer Migration ...Splunk
Presented at SplunkLive! Frankfurt 2018:
Introduction
SIEM Migration Methodology
Use Cases
Datasources & Data Onboarding
ES Architecture
Third-Party Integrations
You Got This!
SplunkLive! Frankfurt 2018 - Get More From Your Machine Data with Splunk AISplunk
Presented at SpluknLive! Frankfurt 2018:
Why AI & Machine Learning?
What is Machine Learning?
Splunk's Machine Learning Tour
Use Cases & Customer Stories
Wrap Up
SplunkLive! Frankfurt 2018 - Data Onboarding OverviewSplunk
Presented at SplunkLive! Frankfurt 2018:
Splunk Data Collection Architecture
Apps and Technology Add-ons
Demos / Examples
Best Practices
Resources and Q&A
SplunkLive! Paris 2018: Use Splunk for Incident Response, Orchestration and A...Splunk
Presented at SplunkLive! Paris 2018:
- Challenges with Security Operations Today
- Overview of Splunk Adaptive Response Initiative
- Technology behind the Adaptive Response Framework
- Demonstrations
- How to build your own AR Action
- Resources
Splunk Discovery: Warsaw 2018 - Reimagining IT with Service IntelligenceSplunk
Presented at Splunk Discovery Warsaw 2018:
What's Service Intelligence and Why You Should Care
Introduction to Splunk IT Service Intelligence
IT Service Intelligence Key Concepts
Demo
Splunk Data Onboarding Overview - Splunk Data Collection ArchitectureSplunk
Splunk's Naman Joshi and Jon Harris presented the Splunk Data Onboarding overview at SplunkLive! Sydney. This presentation covers:
1. Splunk Data Collection Architecture 2. Apps and Technology Add-ons
3. Demos / Examples
4. Best Practices
5. Resources and Q&A
SplunkLive! Amsterdam 2015 Breakout - Getting Started with SplunkSplunk
What is Splunk? At the end of this session you’ll have a high-level understanding of the pieces that make up the Splunk Platform, how it works, and how it fits in the landscape of big data. You’ll see practical examples that differentiate Splunk while demonstrating how to gain quick time to value.
Machine Data Is EVERYWHERE: Use It for TestingTechWell
As more applications are hosted on servers, they produce immense quantities of logging data. Quality engineers should verify that apps are producing log data that is existent, correct, consumable, and complete. Otherwise, apps in production are not easily monitored, have issues that are difficult to detect, and cannot be corrected quickly. Tom Chavez presents the four steps that quality engineers should include in every test plan for apps that produce log output or other machine data. First, test that the data is being created. Second, ensure that the entries are correctly formatted and complete. Third, make sure the data can be consumed by your company’s log analysis tools. And fourth, verify that the app will create all possible log entries from the test data that is supplied. Join Tom as he presents demos including free tools. Learn the steps you need to include in your test plans so your team’s apps not only function but also can be monitored and understood from their machine data when running in production.
Here’s your chance to get hands-on with Splunk for the first time! Bring your modern Mac, Windows, or Linux laptop and we’ll go through a simple install of Splunk. Then, we’ll load some sample data, and see Splunk in action – we’ll cover searching, pivot, reporting, alerting, and dashboard creation. At the end of this session you’ll have a hands-on understanding of the pieces that make up the Splunk Platform, how it works, and how it fits in the landscape of Big Data. You’ll experience practical examples that differentiate Splunk while demonstrating how to gain quick time to value.
Splunk MINT for Mobile Intelligence and Splunk App for Stream for Enhanced Op...Splunk
Learn what is new in Splunk App for Stream and how it can help you utilize wire/network data analytics to proactively resolve applications and IT operational issues and to efficiently analyze security threats in real-time, across your cloud and on-premises infrastructures. Additionally, you will learn about Splunk MINT, which allows you to gain operational intelligence on the availability, performance, and usage of your mobile apps. You’ll learn how to instrument your mobile apps for operational insight, and how you can build the dashboards, alerts, and searches you need to gain real-time insight on your mobile apps.
What is Splunk? At the end of this session you’ll have a high-level understanding of the pieces that make up the Splunk Platform, how it works, and how it fits in the landscape of Big Data. You’ll see practical examples that differentiate Splunk while demonstrating how to gain quick time to value.
Here’s your chance to get hands-on with Splunk for the first time! Bring your modern Mac, Windows, or Linux laptop and we’ll go through a simple install of Splunk. Then, we’ll load some sample data, and see Splunk in action – we’ll cover searching, pivot, reporting, alerting, and dashboard creation. At the end of this session you’ll have a hands-on understanding of the pieces that make up the Splunk Platform, how it works, and how it fits in the landscape of Big Data. You’ll experience practical examples that differentiate Splunk while demonstrating how to gain quick time to value.
What is Splunk? At the end of this session you’ll have a high-level understanding of the pieces that make up the Splunk Platform, how it works, and how it fits in the landscape of Big Data. You’ll see practical examples that differentiate Splunk while demonstrating how to gain quick time to value.
What’s New: Splunk App for Stream and Splunk MINTSplunk
Join us to learn what is new in Splunk App for Stream and how it can help you utilize wire/network data analytics to proactively resolve applications and IT operational issues and to efficiently analyze security threats in real-time, across your cloud and on-premises infrastructures. Additionally, you will learn about Splunk MINT, which allows you to gain operational intelligence on the availability, performance, and usage of your mobile apps. You’ll learn how to instrument your mobile apps for operational insight, and how you can build the dashboards, alerts, and searches you need to gain real-time insight on your mobile apps.
Getting Started with Splunk Enterprise
What is Splunk? At the end of this session you’ll have a high-level understanding of the pieces that make up the Splunk Platform, how it works, and how it fits in the landscape of Big Data. You’ll see practical examples that differentiate Splunk while demonstrating how to gain quick time to value.
Machine-generated data is one of the fastest growing and complex areas of big data. It's also one of the most valuable, containing a definitive record of all user transactions, customer behavior, machine behavior, security threats, fraudulent activity and more. Join us as we explore the basics of machine data analysis and highlight techniques to help you turn your organization’s machine data into valuable insights. This introductory workshop includes a hands-on(bring your laptop) demonstration of Splunk’s technology and covers use cases both inside and outside IT. Learn why more than 13,000 customers in over 110 countries use Splunk to make business, government, and education more efficient, secure, and profitable.
Machine-generated data is one of the fastest growing and complex areas of big data. It's also one of the most valuable, containing some of the most important insights: where things went wrong, how to optimize the customer experience, the fingerprints of fraud. Join us as we explore the basics of machine data analysis and highlight techniques to help you turn your organization’s machine data into valuable insights—across IT and the business. This introductory workshop includes a hands-on (bring your laptop) demonstration of Splunk’s technology and covers use cases both inside and outside IT. Learn why more than 13,000 customers in over 110 countries use Splunk to make their organizations more efficient, secure, and profitable.
Getting Started with Splunk Enterprise Hands-OnSplunk
Here’s your chance to get hands-on with Splunk for the first time! Bring your modern Mac, Windows, or Linux laptop and we’ll go through a simple install of Splunk. Then, we’ll load some sample data, and see Splunk in action – we’ll cover searching, pivot, reporting, alerting, and dashboard creation. At the end of this session, you’ll have a hands-on understanding of the pieces that make up the Splunk Platform, how it works, and how it fits in the landscape of Big Data. You’ll experience practical examples that differentiate Splunk while demonstrating how to gain quick time to value.
What is Splunk? At the end of this session you’ll have a high-level understanding of the pieces that make up the Splunk Platform, how it works, and how it fits in the landscape of Big Data. You’ll see practical examples that differentiate Splunk while demonstrating how to gain quick time to value.
Similar to SplunkLive! Munich 2018: Data Onboarding Overview (20)
.conf Go 2023 - Das passende Rezept für die digitale (Security) Revolution zu...Splunk
.conf Go 2023 presentation:
"Das passende Rezept für die digitale (Security) Revolution zur Telematik Infrastruktur 2.0 im Gesundheitswesen?"
Speaker: Stefan Stein -
Teamleiter CERT | gematik GmbH M.Eng. IT-Sicherheit & Forensik,
doctorate student at TH Brandenburg & Universität Dresden
.conf Go 2023 presentation:
De NOC a CSIRT
Speakers:
Daniel Reina - Country Head of Security Cellnex (España) & Global SOC Manager Cellnex
Samuel Noval - Global CSIRT Team Leader, Cellnex
Splunk - BMW connects business and IT with data driven operations SRE and O11ySplunk
BMW is defining the next level of mobility - digital interactions and technology are the backbone to continued success with its customers. Discover how an IT team is tackling the journey of business transformation at scale whilst maintaining (and showing the importance of) business and IT service availability. Learn how BMW introduced frameworks to connect business and IT, using real-time data to mitigate customer impact, as Michael and Mark share their experience in building operations for a resilient future.
Data foundations building success, at city scale – Imperial College LondonSplunk
Universities have more in common with modern cities than traditional places of learning. This mini city needs to empower its citizens to thrive and achieve their ambitions. Operationalising data is key to building critical services; from understanding complex IT estates for smarter decision-making to robust security and a more reliable, resilient student experience. Juan will share his experience in building data foundations for a resilient future whilst enabling digital transformation at Imperial College London.
Splunk: How Vodafone established Operational Analytics in a Hybrid Environmen...Splunk
Learn how Vodafone has provided end-to-end visibility across services by building an Operational Analytics Platform. In this session, you will hear how Stefan and his team manage legacy, on premise, hybrid and public cloud services, and how they are providing a platform for complex triage and debugging to tackle use cases across Vodafone’s extensive ecosystem.
.italo operates an Essential Service by connecting more than 100 million people annually across Italy with its super fast and secure railway. And CISO Enrico Maresca has been on a whirlwind journey of his own.
Formerly a Cyber Security Engineer, Enrico started at .italo as an IT Security Manager. One year later, he was promoted to CISO and tasked with building out – and significantly increasing the maturity level – of the SOC. The result was a huge step forward for .italo.
So how did he successfully achieve this ambitious ask? Join Enrico as he reveals the key insights and lessons learned in his SOC journey, including:
Top challenges faced in improving security posture
Key KPIs implemented in order to measure success
Strategies and approaches applied in the SOC
How MITRE ATT&CK and Splunk Enterprise Security were utilised
Next steps in their maturity journey ahead
JMeter webinar - integration with InfluxDB and GrafanaRTTS
Watch this recorded webinar about real-time monitoring of application performance. See how to integrate Apache JMeter, the open-source leader in performance testing, with InfluxDB, the open-source time-series database, and Grafana, the open-source analytics and visualization application.
In this webinar, we will review the benefits of leveraging InfluxDB and Grafana when executing load tests and demonstrate how these tools are used to visualize performance metrics.
Length: 30 minutes
Session Overview
-------------------------------------------
During this webinar, we will cover the following topics while demonstrating the integrations of JMeter, InfluxDB and Grafana:
- What out-of-the-box solutions are available for real-time monitoring JMeter tests?
- What are the benefits of integrating InfluxDB and Grafana into the load testing stack?
- Which features are provided by Grafana?
- Demonstration of InfluxDB and Grafana using a practice web application
To view the webinar recording, go to:
https://www.rttsweb.com/jmeter-integration-webinar
Essentials of Automations: Optimizing FME Workflows with ParametersSafe Software
Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place.
Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects.
Here’s what you’ll gain:
- Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows.
- Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy.
- Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency.
- Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity.
We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic.
Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...Ramesh Iyer
In today's fast-changing business world, Companies that adapt and embrace new ideas often need help to keep up with the competition. However, fostering a culture of innovation takes much work. It takes vision, leadership and willingness to take risks in the right proportion. Sachin Dev Duggal, co-founder of Builder.ai, has perfected the art of this balance, creating a company culture where creativity and growth are nurtured at each stage.
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
Connector Corner: Automate dynamic content and events by pushing a buttonDianaGray10
Here is something new! In our next Connector Corner webinar, we will demonstrate how you can use a single workflow to:
Create a campaign using Mailchimp with merge tags/fields
Send an interactive Slack channel message (using buttons)
Have the message received by managers and peers along with a test email for review
But there’s more:
In a second workflow supporting the same use case, you’ll see:
Your campaign sent to target colleagues for approval
If the “Approve” button is clicked, a Jira/Zendesk ticket is created for the marketing design team
But—if the “Reject” button is pushed, colleagues will be alerted via Slack message
Join us to learn more about this new, human-in-the-loop capability, brought to you by Integration Service connectors.
And...
Speakers:
Akshay Agnihotri, Product Manager
Charlie Greenberg, Host
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualityInflectra
In this insightful webinar, Inflectra explores how artificial intelligence (AI) is transforming software development and testing. Discover how AI-powered tools are revolutionizing every stage of the software development lifecycle (SDLC), from design and prototyping to testing, deployment, and monitoring.
Learn about:
• The Future of Testing: How AI is shifting testing towards verification, analysis, and higher-level skills, while reducing repetitive tasks.
• Test Automation: How AI-powered test case generation, optimization, and self-healing tests are making testing more efficient and effective.
• Visual Testing: Explore the emerging capabilities of AI in visual testing and how it's set to revolutionize UI verification.
• Inflectra's AI Solutions: See demonstrations of Inflectra's cutting-edge AI tools like the ChatGPT plugin and Azure Open AI platform, designed to streamline your testing process.
Whether you're a developer, tester, or QA professional, this webinar will give you valuable insights into how AI is shaping the future of software delivery.
5. Basic Architecture Refresh
How Splunk works at a high level
distributed search
auto-load balanced indexing
change tickets
web access logs
windows event logs / perfmon linux logs vmware logs, configs and metrics firewall data
app sever logs jmx and jvm metrics database logs and metrics product pricing
Search Head - Splunk’s UI
Indexer – Data Store/Processing
Forwarder - Collect & Send
Agentless
6. What can Splunk Ingest?
Agent-Less and Forwarder Approach for Flexibility and Optimization
syslog
TCP/UDP
Event Logs, Active Directory, OS Stats
Unix, Linux and Windows hosts
Universal Forwarder
syslog hosts
and network devices
Local File Monitoring
Universal Forwarder
Aggretation
host Windows
Aggregated/API Data Sources
Pre-filtering, API subscriptions
Heavy Forwarder
Mainframes*nix
Wire Data
Splunk Stream
Universal Forwarder or
HTTP Event Collector
DevOps, IoT,
Containers
HTTP Event Collector
(Agentless)
shell
API
perf
7. Collects Data From Remote Sources
• Splunk Universal Forwarders collect data from a local data source and sends it to
one or more Splunk indexers.
Scalable
• Thousands of universal forwarders can be installed with little impact on network
and host performance.
Broad Platform Support
• Available for installation on diverse computing platforms and architectures. Small
computing/disk/memory footprint.
Splunk Universal Forwarder
The Splunk Universal Forwarder is a Separate Download
8. Also Collects Data From Remote Sources...
• ...but is typically used for data aggregation for passage through firewalls, data
routing and/or filtering, scripted/modular inputs, or for HEC endpoints (more on this
in a bit).
Often run as a “data collection node” for API/scripted data access
• A heavy forwarder is typically run as a “data collection node” for technologies
requiring access via API, and not for collection of data from the node itself
Platform Support limited to that of Splunk Enterprise
• Being standalone, Heavy Forwarders are typically run on Linux VMs...
Splunk Heavy Forwarder
Configured via the regular Splunk Enterprise download
9. Large-Scale Data Collection Directly from Applications
• Provides a simple, load-balancer-friendly, secure way (token-based JSON or RAW
API) to send data at scale from applications directly to Splunk
Agentless
• Data at scale can be sent directly to indexer tier, bypassing forwarder layer
Broad Development Platform Support
• Logging drivers available for many platforms (docker, AWS Lambda, etc.) and
simple HTTP endpoint compatible with all development environments
Splunk HTTP Event Collector (HEC)
The Newest Way to Collect Data at Scale
11. App??? Add-on
▶ Your first choice when onboarding
new data
• Clean and ready to go out-of-the-box
▶ App is a complete solution
• Typically uses one or more TAs
▶ Add-on
• Abstracts collection methodology (log file, API,
scripted input, HEC)
• Typically includes relevant field extractions
(schema-on-the-fly)
• Includes relevant config files (props/transforms)
and ancillary scripts binaries
15. ▶ Using the Data Previewer
• Upload a File (You did this in the Getting Started Hands-on Session!)
▶ Installing and using Apps and Add-ons
▶ Continuous Local File Monitoring (Universal Forwarder)
• Monitor a directory and multiple files in real-time
• Most common architecture for syslog-based sourcetypes
What You Will See
17. Components of a Splunk Success Program
Architecture
&
Infrastructure
Operations
& Supporting
Tools
Staffing
Data
On-
Boarding
User
On-Boarding
Inform
18. ▶ Architect
• Design and optimize Splunk architecture for large-scale/distributed
deployments.
▶ System Administrator
• Implement and maintain Splunk infrastructure and configuration
▶ Search Expert
▶ App Developer
▶ Knowledge Manager
• Perform data interpretation, classification and enrichment
• Work with System Administrator to properly onboard data
Typical Splunk Staffing RolesArch &
Infra
Ops &
Tools
Staffing
Data
On-
Boarding
User
On-
Boarding
Inform
19. ▶ Define on-boarding process for
new data sources / apps
▶ Repeatable, documented
process
▶ Provide customer interview
forum or survey
▶ Integrate with service workflow
Data Onboarding TasksArch &
Infra
Ops &
Tools
StaffingData
On-
Boarding
User
On-
Boarding
Inform
New Data Source Request
Provide a data sample
Describe the data’s structure
timestamp | timezone single-/multi-line
sourcetype interesting fields
Describe initial uses for the data
searches | alerts | reports | dashboards
How to collect the data?
UF | syslog | API
How long to retain the data?
Who should have access?
Apply Common information Model
Are there TA’s available?
Validate
20. Ladies and Gentlemen, We’ll be Boarding Soon!
Six Things to Get Right at Index Time
Source
Event
Boundary /
LineBreaking
Host
Index
Sourcetype
Date
Timestamp
21. ▶ Gather info (New Data Source Request):
• Where does this data originate/reside? How will Splunk collect it?
• Which users/groups will need access to this data? Access controls?
• Determine the indexing volume and data retention requirements
• Will this data need to drive existing dashboards (ES, PCI, etc.)?
• Who is the Owner/SME for this data?
▶ Map it out:
• Get a "big enough" sample of the event data
• Identify and map out fields (ensure CIM compliance)
• Assign sourcetype and TA names according to CIM conventions
Pre-Board Essentials
22. ▶ Identify the specific sourcetype(s) - onboard each separately
• Important – syslog is not a sourcetype!
• More on this later
▶ Check for pre-existing app/add-on on splunk.com – don't
reinvent the wheel!
▶ Start with a “Test” index, Verify index-time settings correct
(previous slide)
• Try the Data Previewer first
• tweak props/transforms “by hand” only if absolutely necessary
Pre-Board Essentials (cont.)
23. ▶ Find and fix index-time problems BEFORE
polluting your index
▶ A try-it-before-you-fry-it interface for figuring out
• Event breaking
• Timestamp recognition
• Timezone assignment
▶ Provides most necessary props.conf parameter settings
Your Friend, the Data Previewer
24. If you have to get into the weeds...
Always set these six parameters in props.conf
# SL17
[SL17]
TIME_PREFIX = ^
TIME_FORMAT = %Y-%m-%d %H:%M:%S
MAX_TIMESTAMP_LOOKAHEAD = 19
SHOULD_LINEMERGE = False
LINE_BREAKER = ([nr]+)d{4}-d{2}-d{2}sd{2}:d{2}:d{2}
TRUNCATE = 10000
25. ▶ The Common Information Model (CIM) defines relationships in
the underlying data, while leaving the raw machine data intact
▶ A naming convention for fields, eventtypes & tags
▶ More advanced reporting and correlation requires that the data
be normalized, categorized and parsed
▶ CIM-compliant data sources can drive CIM-based dashboards
(ES, PCI, others)
What Is the CIM and Why Should I Care?
26. ▶ Syslog is a protocol – not a sourcetype
▶ Syslog typically carries multiple sourcetypes
▶ Best to pre-filter syslog traffic using syslog-ng or rsyslog
• Do not send syslog data directly to Splunk over a network port (514)
▶ Use a UF or HEC to transport data to Splunk (next slide)
• Ensures proper load balancing and data distribution
• Secure and efficient
• Insulates against Splunk component failures
▶ See https://www.splunk.com/blog/2017/03/30/syslog-ng-and-hec-scalable-
aggregated-data-collection-in-splunk.html for more info on this topic
A special note on Syslog
29. ▶ https://splunkbase.splunk.com/app/2962/
▶ For creating REST API, Scripted or Modular Inputs through a GUI
▶ Helps your Add-ons get Certified
▶ Can also use on sample data to build out configs as well
Check Out the New Add-on Builder!
30. ▶ Videos!
• http://www.splunk.com/view/education-videos/SP-CAAAGB6
▶ Getting Data In – Splunk Docs
• http://docs.splunk.com/Documentation/Splunk/latest/Data/WhatSplunkcanmonitor
▶ Date and time format variables
• http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/Commontimeformatvariables
▶ Getting Data In – Dev Manual (very thorough!)
• http://dev.splunk.com/view/dev-guide/SP-CAAAE3A
▶ HTTP Event Collector
• http://docs.splunk.com/Documentation/Splunk/latest/Data/UsetheHTTPEventCollector
▶ .conf Sessions
• https://conf.splunk.com/session/2015/conf2015_Aduca_Splunk_Delpoying_OnboardingDataIntoSplunk.pdf
▶ GOOGLE!
Where to Go to Learn More
31. ORLANDO FLORIDA
Walt Disney World Swan and Dolphin Hotels
.conf18:
Monday, October 1 – Thursday, October 4
Splunk University:
Saturday, September 29 – Monday, October 1
Today’s goal is to talk about Data Onboarding or “Getting Data Into Splunk” from a ”New to Splunk” perspective. More specifically we’ll talk about the following and then do a little bit of demo.
1. Splunk Platform – a refresher
You’ve seen the Splunk Overview, but I want to quickly go through a few overview slides and relate why data onboarding is important to them
2. What can Splunk Eat
Then we’ll identify not only the data sources that Splunk can collect, but the methods of collection as well
3. Apps and Add-ons
Next we’ll discuss how Apps and Add-ons from the ecosystem play a role
4. Data Onboarding Examples/Demos
We’ll get into a few demos
5. Data Onboarding Best Practices and Next Steps
And finally we’ll get into some common best practices and what to do from here!
1. Explain the different components at a high level
2. The forwarder is one of the many ways to collect data in Splunk – we will discuss setting up and using a forwarder in more detail later in the presentation
1. Spend some time talking about each collection method
2. Today we will concentrate on and demo the ones highlighted in blue
Universal Forwarders provide reliable, secure data collection from remote sources and forward that data into Splunk software for indexing and consolidation. They can scale to tens of thousands of remote systems, collecting terabytes of data.
Heavy forwarders allow for the aggregation, filtering and routing of data, as well as serving as a “data collection node” for applications such as DB Connect and other API-driven data sources. They are typically *not* used for local data collection.
HTTP Event Collector (HEC pronounced H-E-C) is a new, robust, token-based JSON/raw API for sending events to Splunk from anywhere without requiring a forwarder. It is designed for performance and scale. Using a load balancer in front, it can be deployed to handle millions of events per second. It is highly available and it is secure. It is easy to configure, easy to use, and best of all it works out of the box.* A few other cool tidbits, it supports gzip compression, batching, HTTP keep-alive and HTTP/HTTPs.
Splunk apps and add-ons: what & why?
Splunk apps allow developers to extend data ingestion and processing capabilities of Splunk Enterprise for your specific needs. Apps facilitate more efficient completion of domain-specific tasks by the end user.
High-level perspective
A Splunk app is a prebuilt collection of additional capabilities packaged for a specific technology, or use cases, which allows a more effective usage of Splunk Enterprise. You can use Splunk apps to gain the specific insights you need from your machine data.
Depending on the type and complexity of those use cases, and also whether the developer wants certain app parts to be configured or distributed separately (potentially by a third party), an app may rely on various add-ons.
An add-on is a technical component that can be re-used across a number of different use cases and packaged with one or more Splunk apps. Add-ons may contain one or more knowledge objects, which encapsulate a specific functionality focused on a single concern and its configuration. Using an add-on should help to reduce the technical risk and cost of building an app.
Additionally we have the community!
The community provides thousands of apps and add-ons that can help you onboard and ingest thousands of different data types and new content is added everyday!
Let’s look at how we would use an Add-on from Splunkbase to get data in.
Use an example that you are comfortable with and showcases using an add-on to get data in and mapped properly.
< If you have another data source or want to improvise a little here feel free – otherwise you can use the following demo flow below >
< Support files can be found here: LINK >
1. Install an instance of Splunk on your laptop.
2. Create an inputs.conf that monitors a directory that will contain the PANW logs files, using the PANW sourcetype from the TA. Leave the directory empty for now.
3. Show the data preview wizard with the apache data. Show how Splunk understands (and assigns an appropriate sourcetype) to the data. Show proper field extractions when ingest is complete.
4. Use the wizard to upload one of the 5 PANW data files. Show how the sourcetype is *not* automatically set, and that there are no relevant choices in the sourcetype picker in the Wizard. Set the sourcetype to some arbitrary value. Show that there are no relevant field extractions after ingest.
5. Now, install the PANW app. Make sure to RESTART.
6. Use the wizard to import the next PANW data file. Now show that there *is* a relevant sourcetype in the picker. Select it.
7. Show how fields are extracted properly in the data. *HOWEVER* -- note that the original sourcetype is automatically changed by the TA, and you will get no results when jumping from the wizard into the search window. Instead, show the 5 or 6 new sourcetypes that get generated as a result of the TA doing its thing.
8. Lastly, deposit the 3rd PANW data file into the monitor directory set up earlier. Show the data in search, correctly sourcetyped.
9. Move the file to a “backup” filename in the monitored directory. Show how Splunk does *not* reingest the data.
10. Add the 4th PANW data sample. Show how the UF handles this.
In this next section we are only looking at the tip of the iceberg. Data Onboarding can quickly become an advanced topic so the point of this next session is to introduce you to some of the most important/key points to get you started. After that you’ll need to do some research and learn the specifics yourself.
These are the components that make up a successful Splunk program – both large and small. In a very large deployment, individual people (or more) can be dedicated to each of these components.
Appropriate staffing will ensure these components are properly addressed. The person responsible for data onboarding from an architectural perspective is the Knowledge Manager
It is important to have a defined, documented, and repeatable process for data onboarding.
Explain Index Time
Spend some time saying why these are so important for Splunk. Mention there will be references and resources at the end of the presentation to help dive deeper into these topics.
It is important to not only get the technical details right, but also the data stewardship issues: Who owns the data, who can see it, and how long to keep it?
It is important to “do the homework” prior to onboarding, not only to get the index-time parameters correct (previous slide) but also to ensure the resultant data in Splunk will be of value to the widest variety of people and use cases
Make sure to show this in the demo, this slide is just a follow up reminder
These are min number of parameters, that should be set when creating a new data source. Again, I like said when I flashed up the Splunk Apps site.. Find something similar to your source and re-work it. But make sure it includes these parameters.
Normalizes data from different sources – Host and hostname discussion
Syslog represents almost 50% of a typical Splunk installation’s data. And yet syslog itself is simply the protocol over which a number of devices’ log data flows. Be sure to *not* use syslog as the sourcetype, but rather that of the originating data. Use appropriate syslog tools to pre-filter data. They’re good at it, they’re free, they’re well-documented, and they integrate well with Splunk.
HEC is the newest, and most scalable, way to collect syslog-based data.
In addition to live, .conf, docs, answers, meetups etc etc
Don’t forget to complete today’s survey at ponypoll.com/______ for your chance to win a .conf2017 pass. (Make sure you put the right PonyPoll link!)
A winner will be identified tomorrow through a random drawing from completed surveys and will be notified via email.