The document discusses how Splunk can help users manage and analyze Internet of Things (IoT) data. Splunk provides tools to collect data from various sources, search and correlate the data, and build applications and visualizations. This allows users to harness IoT data from devices, sensors, and industrial systems. Splunk also offers developer tools like APIs and SDKs to build custom IoT applications on its platform.
GAINING APPLICATION LIFECYCLE INTELLIGENCE
Applied Spring Track
Today we are facing an ever-increasing speed of product delivery. DevOps practices
like continuous integration and deployment increase the dependence of systems
like task tracking and source code repositories with build servers and test suites.
With data moving rapidly through these different tools, it becomes challenging to
maintain a grasp of the process, especially as the data is distributed and in a variety
of formats. But it is still critical to maintain full visibility of the product development
journey – from user stories to production data. By starting at the beginning of the
Product Development Lifecycle, you can track a problem in production all the way
back to the code that was checked into the build and the developer responsible for
the code.
In this session I'll demonstrate some of the ways in which Splunk software can be
used to collect and correlate data throughout the various stages of the lifecycle of
your code, to ultimately make you more efficient and make your code better.
Splunk is a powerful platform that can harness your machine data and turn it into valuable information thereby enabling your business to make informed decisions, taking your organization from reactive to proactive. Just like any other platform, Splunk is only as powerful as the data it has access to, therefore in this session we will be conducting a walk thru of how to successfully on-board data, with samples of data ranging from simple to complex. We will also be taking a look at how to use common TA’s to bring valuable data into Splunk. This session is designed to give you a better understanding of how to onboard data into Splunk enabling you to unlock the power of your data.
GAINING APPLICATION LIFECYCLE INTELLIGENCE
Applied Spring Track
Today we are facing an ever-increasing speed of product delivery. DevOps practices
like continuous integration and deployment increase the dependence of systems
like task tracking and source code repositories with build servers and test suites.
With data moving rapidly through these different tools, it becomes challenging to
maintain a grasp of the process, especially as the data is distributed and in a variety
of formats. But it is still critical to maintain full visibility of the product development
journey – from user stories to production data. By starting at the beginning of the
Product Development Lifecycle, you can track a problem in production all the way
back to the code that was checked into the build and the developer responsible for
the code.
In this session I'll demonstrate some of the ways in which Splunk software can be
used to collect and correlate data throughout the various stages of the lifecycle of
your code, to ultimately make you more efficient and make your code better.
Splunk is a powerful platform that can harness your machine data and turn it into valuable information thereby enabling your business to make informed decisions, taking your organization from reactive to proactive. Just like any other platform, Splunk is only as powerful as the data it has access to, therefore in this session we will be conducting a walk thru of how to successfully on-board data, with samples of data ranging from simple to complex. We will also be taking a look at how to use common TA’s to bring valuable data into Splunk. This session is designed to give you a better understanding of how to onboard data into Splunk enabling you to unlock the power of your data.
What is Splunk? At the end of this session you’ll have a high-level understanding of the pieces that make up the Splunk Platform, how it works, and how it fits in the landscape of Big Data. You’ll see practical examples that differentiate Splunk while demonstrating how to gain quick time to value.
Using Metrics for Fun, Developing with the KV Store + Javascript & News from ...Harry McLaren
We explore "Metrics, mstats and Me: Splunking Human Data” and also have some insights into the KV Store and javascript use in dashboards. We’ll also re-cover the conf18 updates for those who couldn’t attend our last session.
Webinar: Improve Splunk Analytics and Automate Processes with SnapLogicSnapLogic
Last week SnapLogic sponsored partner event Splunk Worldwide Users' Conference in Las Vegas. The theme of the conference was "Your Data, No Limits." In keeping with this theme, SnapLogic helps Splunk customers access more comprehensive analytics by integrating as much data as possible from as many sources as possible, and by streamlining the business process of loading data in Splunk, detecting problems, and facilitating actions that result in a prompt resolution.
To learn more, visit: http://www.snaplogic.com/.
Covering off some of the latest announcements at Splunk's user conference (.conf), an Add-on created to Splunk config files and also the presentation delivered at .conf18 on SplDevOps!
Apache Flink: Real-World Use Cases for Streaming AnalyticsSlim Baltagi
This face to face talk about Apache Flink in Sao Paulo, Brazil is the first event of its kind in Latin America! It explains how Apache Flink 1.0 announced on March 8th, 2016 by the Apache Software Foundation (link), marks a new era of Big Data analytics and in particular Real-Time streaming analytics. The talk maps Flink's capabilities to real-world use cases that span multiples verticals such as: Financial Services, Healthcare, Advertisement, Oil and Gas, Retail and Telecommunications.
In this talk, you learn more about:
1. What is Apache Flink Stack?
2. Batch vs. Streaming Analytics
3. Key Differentiators of Apache Flink for Streaming Analytics
4. Real-World Use Cases with Flink for Streaming Analytics
5. Who is using Flink?
6. Where do you go from here?
IoT Edge Processing with Apache NiFi and MiniFi and Apache MXNet for IoT NY 2018Timothy Spann
IoT Edge Processing with Apache NiFi and MiniFi and Apache MXNet for IoT NY 2018. A quick talk on how to ingest IoT sensor data, camera images and run deep learning prebuilt models on edge devices including Raspberry Pis and NVidia Jetson TX1s. From there data is processed and orchestrated with Apache NiFi to send to various Big Data backends.
What is Splunk? At the end of this session you’ll have a high-level understanding of the pieces that make up the Splunk Platform, how it works, and how it fits in the landscape of Big Data. You’ll see practical examples that differentiate Splunk while demonstrating how to gain quick time to value.
Using Metrics for Fun, Developing with the KV Store + Javascript & News from ...Harry McLaren
We explore "Metrics, mstats and Me: Splunking Human Data” and also have some insights into the KV Store and javascript use in dashboards. We’ll also re-cover the conf18 updates for those who couldn’t attend our last session.
Webinar: Improve Splunk Analytics and Automate Processes with SnapLogicSnapLogic
Last week SnapLogic sponsored partner event Splunk Worldwide Users' Conference in Las Vegas. The theme of the conference was "Your Data, No Limits." In keeping with this theme, SnapLogic helps Splunk customers access more comprehensive analytics by integrating as much data as possible from as many sources as possible, and by streamlining the business process of loading data in Splunk, detecting problems, and facilitating actions that result in a prompt resolution.
To learn more, visit: http://www.snaplogic.com/.
Covering off some of the latest announcements at Splunk's user conference (.conf), an Add-on created to Splunk config files and also the presentation delivered at .conf18 on SplDevOps!
Apache Flink: Real-World Use Cases for Streaming AnalyticsSlim Baltagi
This face to face talk about Apache Flink in Sao Paulo, Brazil is the first event of its kind in Latin America! It explains how Apache Flink 1.0 announced on March 8th, 2016 by the Apache Software Foundation (link), marks a new era of Big Data analytics and in particular Real-Time streaming analytics. The talk maps Flink's capabilities to real-world use cases that span multiples verticals such as: Financial Services, Healthcare, Advertisement, Oil and Gas, Retail and Telecommunications.
In this talk, you learn more about:
1. What is Apache Flink Stack?
2. Batch vs. Streaming Analytics
3. Key Differentiators of Apache Flink for Streaming Analytics
4. Real-World Use Cases with Flink for Streaming Analytics
5. Who is using Flink?
6. Where do you go from here?
IoT Edge Processing with Apache NiFi and MiniFi and Apache MXNet for IoT NY 2018Timothy Spann
IoT Edge Processing with Apache NiFi and MiniFi and Apache MXNet for IoT NY 2018. A quick talk on how to ingest IoT sensor data, camera images and run deep learning prebuilt models on edge devices including Raspberry Pis and NVidia Jetson TX1s. From there data is processed and orchestrated with Apache NiFi to send to various Big Data backends.
Splunk for Industrial Data and the Internet of ThingsSplunk
Splunk software provides a scalable and versatile platform for machine data generated by all of the devices, control systems, sensors, SCADA, networks, applications and end users connected by today's networks. In this session we will discuss and demo how you can use Splunk software to gain insights into machine data generated by devices and control systems. We’ll cover common themes in use cases, and show you how to access the free apps and add-ons that simplify the connection and collection of data from both industrial systems and the Internet of Things. In addition we will introduce you to Splunk’s growing ecosystem of IoT and Industrial focused technology partners.
Find out how Vertu has transformed on three fronts: Cloud, DevOps and Data over the last 5 years and in particular how we make use of Splunk. This talk was given at Gartner Catalyst 2015 in London, as well as at IP Expo Europe 2015.
What is Splunk? At the end of this session you’ll have a high-level understanding of the pieces that make up the Splunk Platform, how it works, and how it fits in the landscape of Big Data. You’ll see practical examples that differentiate Splunk while demonstrating how to gain quick time to value.
Here’s your chance to get hands-on with Splunk for the first time! Bring your modern Mac, Windows, or Linux laptop and we’ll go through a simple install of Splunk. Then, we’ll load some sample data, and see Splunk in action – we’ll cover searching, pivot, reporting, alerting, and dashboard creation. At the end of this session you’ll have a hands-on understanding of the pieces that make up the Splunk Platform, how it works, and how it fits in the landscape of Big Data. You’ll experience practical examples that differentiate Splunk while demonstrating how to gain quick time to value.
Using ATL/EMFTVM for import/export of medical data - #sda2014Dennis Wagelaar
Most datacentric software must deal with some form of import/export of their internal data model to an external data format. In many cases, this external data format is some sort of standard format, or otherwise dictated by external sources, and does not map one-on-one to the internal data model. This was also the case for CareConnect, HealthConnect/Corilus‘ latest Electronic Medical Record software. CareConnect must be able to import/export its data from/to SUMEHR, PMF, and GPSMF documents. On top of this comes that CareConnect’s internal data model consists of some 300 classes, which means there are a lot of mappings to define. To deal with the size and complexity of this scenario, we decided to use a specialised language: the ATL transformation language in combination with the EMF Transformation Virtual Machine: EMFTVM is a new runtime for ATL, which adds a number of performanceenhancing features that make it more suitable for use within a Java application. The declarative, rulebased nature of ATL allowed us to write more concise code as well as distribute the workload of writing the ATL transformation code over multiple developers. This significantly increased our ability to deal with complexity.
Travis Perkins: Building a 'Lean SOC' over 'Legacy SOC'Splunk
Travis Perkins has a complex hybrid IT infrastructure and is in midst of migrating to the cloud. This session will outline the pitfalls from their initial infrastructure-heavy ‘legacy SOC’ approach with a legacy SIEM and the success they gained when they moved to a cloud-based, data-driven ‘lean SOC’.
SplunkLive! Amsterdam 2015 Breakout - Getting Started with SplunkSplunk
What is Splunk? At the end of this session you’ll have a high-level understanding of the pieces that make up the Splunk Platform, how it works, and how it fits in the landscape of big data. You’ll see practical examples that differentiate Splunk while demonstrating how to gain quick time to value.
Supporting Enterprise System Rollouts with SplunkErin Sweeney
At Cricket Communications, Splunk started as a way to correlate all of our data into one view to help our operations team keep processes humming. Then we gave secured access to our developers, now they’re addicted. In fact, Splunk is critical in helping us speedup deployment of new systems (like our recent multi-million dollar billing system implementation). Learn how we use Splunk to display key metrics for the business, track overall system health, track transactions, optimize license usage, and support capacity
planning.
What is Splunk? At the end of this session you’ll have a high-level understanding of the pieces that make up the Splunk Platform, how it works, and how it fits in the landscape of Big Data. You’ll see practical examples that differentiate Splunk while demonstrating how to gain quick time to value.
What is Splunk? At the end of this session you’ll have a high-level understanding of the pieces that make up the Splunk Platform, how it works, and how it fits in the landscape of Big Data. You’ll see practical examples that differentiate Splunk while demonstrating how to gain quick time to value.
Similar to QCon London 2015 - Wrangling Data at the IOT Rodeo (20)
Explore our comprehensive data analysis project presentation on predicting product ad campaign performance. Learn how data-driven insights can optimize your marketing strategies and enhance campaign effectiveness. Perfect for professionals and students looking to understand the power of data analysis in advertising. for more details visit: https://bostoninstituteofanalytics.org/data-science-and-artificial-intelligence/
Techniques to optimize the pagerank algorithm usually fall in two categories. One is to try reducing the work per iteration, and the other is to try reducing the number of iterations. These goals are often at odds with one another. Skipping computation on vertices which have already converged has the potential to save iteration time. Skipping in-identical vertices, with the same in-links, helps reduce duplicate computations and thus could help reduce iteration time. Road networks often have chains which can be short-circuited before pagerank computation to improve performance. Final ranks of chain nodes can be easily calculated. This could reduce both the iteration time, and the number of iterations. If a graph has no dangling nodes, pagerank of each strongly connected component can be computed in topological order. This could help reduce the iteration time, no. of iterations, and also enable multi-iteration concurrency in pagerank computation. The combination of all of the above methods is the STICD algorithm. [sticd] For dynamic graphs, unchanged components whose ranks are unaffected can be skipped altogether.
Show drafts
volume_up
Empowering the Data Analytics Ecosystem: A Laser Focus on Value
The data analytics ecosystem thrives when every component functions at its peak, unlocking the true potential of data. Here's a laser focus on key areas for an empowered ecosystem:
1. Democratize Access, Not Data:
Granular Access Controls: Provide users with self-service tools tailored to their specific needs, preventing data overload and misuse.
Data Catalogs: Implement robust data catalogs for easy discovery and understanding of available data sources.
2. Foster Collaboration with Clear Roles:
Data Mesh Architecture: Break down data silos by creating a distributed data ownership model with clear ownership and responsibilities.
Collaborative Workspaces: Utilize interactive platforms where data scientists, analysts, and domain experts can work seamlessly together.
3. Leverage Advanced Analytics Strategically:
AI-powered Automation: Automate repetitive tasks like data cleaning and feature engineering, freeing up data talent for higher-level analysis.
Right-Tool Selection: Strategically choose the most effective advanced analytics techniques (e.g., AI, ML) based on specific business problems.
4. Prioritize Data Quality with Automation:
Automated Data Validation: Implement automated data quality checks to identify and rectify errors at the source, minimizing downstream issues.
Data Lineage Tracking: Track the flow of data throughout the ecosystem, ensuring transparency and facilitating root cause analysis for errors.
5. Cultivate a Data-Driven Mindset:
Metrics-Driven Performance Management: Align KPIs and performance metrics with data-driven insights to ensure actionable decision making.
Data Storytelling Workshops: Equip stakeholders with the skills to translate complex data findings into compelling narratives that drive action.
Benefits of a Precise Ecosystem:
Sharpened Focus: Precise access and clear roles ensure everyone works with the most relevant data, maximizing efficiency.
Actionable Insights: Strategic analytics and automated quality checks lead to more reliable and actionable data insights.
Continuous Improvement: Data-driven performance management fosters a culture of learning and continuous improvement.
Sustainable Growth: Empowered by data, organizations can make informed decisions to drive sustainable growth and innovation.
By focusing on these precise actions, organizations can create an empowered data analytics ecosystem that delivers real value by driving data-driven decisions and maximizing the return on their data investment.
Chatty Kathy - UNC Bootcamp Final Project Presentation - Final Version - 5.23...John Andrews
SlideShare Description for "Chatty Kathy - UNC Bootcamp Final Project Presentation"
Title: Chatty Kathy: Enhancing Physical Activity Among Older Adults
Description:
Discover how Chatty Kathy, an innovative project developed at the UNC Bootcamp, aims to tackle the challenge of low physical activity among older adults. Our AI-driven solution uses peer interaction to boost and sustain exercise levels, significantly improving health outcomes. This presentation covers our problem statement, the rationale behind Chatty Kathy, synthetic data and persona creation, model performance metrics, a visual demonstration of the project, and potential future developments. Join us for an insightful Q&A session to explore the potential of this groundbreaking project.
Project Team: Jay Requarth, Jana Avery, John Andrews, Dr. Dick Davis II, Nee Buntoum, Nam Yeongjin & Mat Nicholas
As Europe's leading economic powerhouse and the fourth-largest hashtag#economy globally, Germany stands at the forefront of innovation and industrial might. Renowned for its precision engineering and high-tech sectors, Germany's economic structure is heavily supported by a robust service industry, accounting for approximately 68% of its GDP. This economic clout and strategic geopolitical stance position Germany as a focal point in the global cyber threat landscape.
In the face of escalating global tensions, particularly those emanating from geopolitical disputes with nations like hashtag#Russia and hashtag#China, hashtag#Germany has witnessed a significant uptick in targeted cyber operations. Our analysis indicates a marked increase in hashtag#cyberattack sophistication aimed at critical infrastructure and key industrial sectors. These attacks range from ransomware campaigns to hashtag#AdvancedPersistentThreats (hashtag#APTs), threatening national security and business integrity.
🔑 Key findings include:
🔍 Increased frequency and complexity of cyber threats.
🔍 Escalation of state-sponsored and criminally motivated cyber operations.
🔍 Active dark web exchanges of malicious tools and tactics.
Our comprehensive report delves into these challenges, using a blend of open-source and proprietary data collection techniques. By monitoring activity on critical networks and analyzing attack patterns, our team provides a detailed overview of the threats facing German entities.
This report aims to equip stakeholders across public and private sectors with the knowledge to enhance their defensive strategies, reduce exposure to cyber risks, and reinforce Germany's resilience against cyber threats.
4. The cowboy metaphor
4
Data wrangling / lassoing (capturing)
Data needs harnessing (bring under control for analysis)
Data might need a little grooming (clean, filter)
Data might need branding (categorizing / labeling / enrichment)
Data corralling (correlation)
Data stabling (securing)
Data needs to go to the rodeo (a platform)
Make data useful = be a data cowboy
11. Operational Technology
Energy Oil & Gas Process Buildings Mfg
Transport-
ation
Medical
Devices
Telecom
Consumer
Technology
Smart
Home
Wearables Media
The landscape is much, much vaster
Industrial Data Producing Assets
12. Succeeding with IOT data
12
IOT data is already being generated
And we are already capturing this data
The key challenge will be in turning this into something genuinely
useful. This is the opportunity.
Enable the developers & data domain experts
Give them the platforms and tools to be productive
This leads to ECOSYSTEM
14. Splunk can help you become an IOT data cowboy
14
Wrangle – Collect the data
Harness – Search over the data / Correlate
Show at the Rodeo – Visualize the data/Alerting
Provide a platform for Developers to build IOT Apps
15. Platform for machine data
15
Splunk storage Other Big Data stores
Developer
Platform
Data collection
and indexing
Report
and
analyze
Custom
dashboards
Monitor
and alert
Ad hoc
search
20. Kepware Industrial Data Forwarder for Splunk
20
Real-time streaming data collection from 150+ industrial protocolsProprietary and legacy data translation
24. Some data from a work order
05/27/2014T10:24:17GMT applicationId="safetyObs" eventType="safety" assetID="CV1002384-1045"
employeeId="114635" jobSite="PLEC-2014-GC" observationId="184568-451124-256" observation="Control Valve handle
extracted to manual position. No lockout/tagout or other tag visible. Process is running." observationCriticality="5"
imageId="PLEC-2014-GC-184568-451124-256" imageUri="https://mybucket.s3.amazonaws.com/PLEC-2014-GC-184568-
451124-256.png"
1543541, workorder, bsic, 78544, pipefitting, CV1002384, "install manual bleed bypass", 04/13/2014, 05/21/2014, 25663,
complete
05/22/2014 03:17:31 asset_id="CV1002384-1045" process_id="batch transfer starting" alarm="control valve failed to open"
05/22/2014 04:21:45 asset_id="CV1002384-1045" process_id="batch transfer starting" alarm="control valve failed to open"
05/22/2014 06:35:39 asset_id="CV1002384-1045" process_id="batch transfer starting" alarm="control valve failed to open"
05/22/2014 07:40:29 asset_id="CV1002384-1045" process_id="batch transfer starting" alarm="control valve failed to open"
CMMS (Work Order)
Application
29. Splunk options
29
Splunk> Enterprise : Free to download and use. Index 500 MB/day.
Splunk> Cloud : Premium, cloud hosted. Full Enterprise stack.100% uptime.
Splunk> Sandbox : Spin up a cloud instance in minutes. Load in data.
Hunk> : Splunk for data in Hadoop HDFS, MongoDB , other datastores (Neo4J)
IOT is the buzzword of the moment. And it is warranted. An internet of things is actually starting to exist and is growing at an exponential pace. But this brings with it new and exciting challenges.And by that I mean how are we going to capture the huge volume and variety of IOT data and turn it into something genuinely useful for the end user.So in this talk I'll put my cowboy hat on and and show a few ways in which you can use Splunk to harness all this data and create compelling IOT apps . Yee Ha !
Cowboy metaphor is very applicable to data , it’s about the end to end iterative lifecycle of data.
This is what we are going to talk about today , a few marketing slides , we are sponsors
Actually show how to make something useful with IOT data , yes we will make something !!!
Lets find some ways to become data cowboys today.
Stampede , Spurring , Ranch , Saddle , Reins , sure there are other metaphors , email your suggestions
[We hear about Big Data from IT, but scope is ever-widening as more devices become internet-enabled]
Data is generated everywhere
It’s not just the servers, applications, network devices, security devices in your datacenters.
Planes, trains, cars – all generate data.
The mobile device in your pocket – is continuously generating data even as you sit there.
Industrial control system (ICS), which had little resemblance to traditional information technology (IT) systems (isolated systems running proprietary control protocols using specialized hardware and software) are being replaced by Internet-connected devices – opening up opportunity and the the possibility of cyber security vulnerabilities and incidents.
Staggering scale=staggering analytics
The scale of all of this is staggering, and it is going to take machine data platforms for those who are responsible for the production, maintenance, management of these devices to handle this type of scale without giving up the additional insight real-time analytics provides.
2014 IDC report – “Universe of data is doubling in size every two years – by 2020 containing nearly as many digital bits as there are stars in the universe. 44 trillion gigabytes”
New challenges , new approaches needed
We are not living in an ETL landscape anymore , the lay of the land is different and we need different approaches.
[This machine data is high volume, diverse and very valuable, containing a record of all activity and behavior – it’s hard or impossible to process and analyze for traditional approaches or in a timely manner]
Machine data is one of the fastest growing, most complex and most valuable segments of data.
Why? it contains a categorical record of activity and behavior – of your customers, your applications, servers, sensors, devices, and so on.
Characteristics of machine data – the four V’s - the last two are the most interesting / challenging.
Want to talk a bit about this particular source of data, Im’ most interested in this
Devices are now joining the network at pace we have not seen before , not necessarily emiiting more data , just more devices
Growing at a massive pace and represents the greatest domain of opportunity
Just a natural evolution of the delivery of information over the internet
What are folks saying and doing
Devices / data volume
Investment
Opportunity for business
Apis for developers
A vibrant ecosystem of opportunity
From Cisco:
A jet engine generates 1TB of data per flight.
A large refinery generates 1TB of raw data per day.
As cars get smarter, the number of sensors is projected to reach as many as 200 per car.
Sensors of all types will generate immense amounts of data. In fact, analysts estimate that by 2020, 40 percent of all data will come from sensors.
Not just your fitbit , washing machine or connected bed
There are two distinct segments of the IoT—the largely siloed, enterprise-managed industrial systems, consisting of legacy applications and heavy equipment (e.g., oil and gas, power distribution and manufacturing systems); and consumer-focused, cloud-connected devices like connected vending machines, wearables and smarthome appliances.
Whether in industrial systems or the consumer-focused IoT, the constant processes, applications, interactions, sensor readings, geolocation information and streams of data can be easily managed and analyzed using Splunk software.
Smart buildings , elevators
Smart Car
Healthcare devices
But the challenges are stil the same
There is no point in just capturing and storing IOT data , that is not being productive or pragmatic
You have to unleash it into something of use
Create compelling apps
A thriving developer ecosystem will evolve ideas and uses cases for you , you just have to give them the platform to suceed.
Other challenges like security and scalability and timely processing
Platform for collecting data from any source , in any format
Search over it , correlate , look for insights
Visualize it , build apps for your domain use case
Splunk’s flagship product is Splunk Enterprise. Splunk Enterprise is a fully featured, powerful platform for collecting, searching, monitoring and analyzing machine data.
Splunk collects machine data securely and reliably from wherever it’s generated. It stores and indexes the data in real time in a centralized location and protects it with role-based access controls. You can even leverage other data stores. Splunk lets you search, monitor, report and analyze your real-time and historical data. Now you have the ability to quickly visualize and share your data, no matter how unstructured, large or diverse it may be.
Troubleshoot problems and investigate security incidents in minutes (not hours or days). Monitor your end-to-end infrastructure to avoid service degradation or outages. Gain real-time visibility and critical insights into customer experience, transactions and behavior. Use Splunk and make your data accessible, usable and valuable across the enterprise.
Splunk collects and indexes any machine data from virtually any source, format or location in real time. This includes data streaming from packaged and custom applications, app servers, web servers, databases, networks, virtual machines, telecoms equipment, OS’s, sensors, and much more. There’s no requirement to “understand” the data upfront. Just point Splunk at your data or deploy Splunk forwarders to reliably stream data from remote systems at scale. Splunk immediately starts collecting and indexing, so you can start searching and analyzing.
No more armies of consultants, or a DBA to make it work.
Connection to industrial devices and applications and translation of proprietary industrial protocols is outside the scope of our technology
Some customers and partners have developed custom connectors or have leveraged existing middleware applications, but this approach has a high barrier to entry and does not allow fast time to value or much flexibility after the initial integration
We need to align with technology that is proven and aligns with our own scalability, security, and flexibility.
Through our partnership with Kepware, Splunk can now ingest data from thousands of devices communicating over more than 150 proprietary and legacy industrial protocols.
Let’s take a peek at what machine data looks like.
Safety is #1 priority for you and in this example, we see here excerpts from 3 typically siloed operational technology systems:
Safety Observation Reporting System
Computerized Maintenance and Management System (CMMS)
Alarm Logs from the Plant Control or SCADA system.
Some machine data contains human-generated content. For example this logfile from a safety app on a technicians mobile device. The application allows them to easily report either positive, negative or neutral safety observations on a regular basis.
(logfile – timestamp, app_id, etc.)
Here we see fields from a work order application. This is relational data from a relational database backed work order management system like MAXIMO. Fields include work_order_id, the technical_id that performed the work and the date the work order was created and completed.
Here is a repeating alarm from the SCADA system – oftentimes repeating alarms such as this are considered a nuisance without the context of … <next slide>
Alerted in Splunk by a level 5 “Critical” safety observation entering the safety observation system.
Splunk is able to extract the asset identification automatically from the observation, and using the Splunk app DBConnect, correlate information from the CMMS (Workorder) application’s relational database back end, which can be used to supplement the observation with information such as the technician who performed the work (for mitigation through additional training), and the date the work was completed.
This complete date can then be used to automatically search Splunk’s historical record of alarms and events for this particular asset ID, and show that operations were negatively impacted by the issue.
In the future, repeating alarms of this type could automatically trigger an investigation using the details provided in the observation, and troubleshooting could be greatly accelerated. In fact, using scripted alerts in Splunk, the workorder to investigate the valve handle position could be automatically created and dispatched.
Splunk provides a single pane of glass view into comprehensive facility operations unmatched by other specialized facility or factory ops systems.
BUILD SPLUNK APPS
The Splunk Web Framework makes building a Splunk app looks and feels like building any modern web application.
The Simple Dashboard Editor makes it easy to BUILD interactive dashboards and user workflows as well as add custom styling, behavior and visualizations. Simple XML is ideal for fast, lightweight app customization and building. Simple XML development requires minimal coding knowledge and is well-suited for Splunk power users in IT to get fast visualization and analytics from their machine data. Simple XML also lets the developer “escape” to HTML with one click to do more powerful customization and integration with JavaScript.
Developers looking for more advanced functionality and capabilities can build Splunk apps from the ground up using popular, standards-based web technologies: JavaScript and Django. The Splunk Web Framework lets developers quickly create Splunk apps by using prebuilt components, styles, templates, and reusable samples as well as supporting the development of custom logic, interactions, components, and UI. Developers can choose to program their Splunk app using Simple XML, JavaScript or Django (or any combination thereof).
EXTEND AND INTEGRATE SPLUNK
Splunk Enterprise is a robust, fully-integrated platform that enables developers to INTEGRATE data and functionality from Splunk software into applications across the organization using Software Development Kits (SDKs) for Java, JavaScript, C#, Python, PHP and Ruby. These SDKs make it easier to code to the open REST API that sits on top of the Splunk Engine. With almost 200 endpoints, the REST API lets developers do programmatically what any end user can do in the UI and more. The Splunk SDKs include documentation, code samples, resources and tools to make it faster and more efficient to program against the Splunk REST API using constructs and syntax familiar to developers experienced with Java, Python, JavaScript, PHP, Ruby and C#. Developers can easily manage HTTP access, authentication and namespaces in just a few lines of code.
Developers can use the Splunk SDKs to:
- Run real-time searches and retrieve Splunk data from line-of-business systems like Customer Service applications
- Integrate data and visualizations (charts, tables) from Splunk into BI tools and reporting dashboards
- Build mobile applications with real-time KPI dashboards and alerts powered by Splunk
- Log directly to Splunk from remote devices and applications via TCP, UDP and HTTP
- Build customer-facing dashboards in your applications powered by user-specific data in Splunk
- Manage a Splunk instance, including adding and removing users as well as creating data inputs from an application outside of Splunk
- Programmatically extract data from Splunk for long-term data warehousing
Developers can EXTEND the power of Splunk software with programmatic control over search commands, data sources and data enrichment.
Splunk Enterprise offers search extensibility through:
- Custom Search Commands - developers can add a custom search script (in Python) to Splunk to create own search commands. To build a search that runs recursively, developers need to make calls directly to the REST API
- Scripted Lookups: developers can programmatically script lookups via Python.
- Scripted Alerts: can trigger a shell script or batch file (we provide guidance for Python and PERL).
- Search Macros: make chunks of a search reuseable in multiple places, including saved and ad hoc searches.
Splunk also provides developers with other mechanisms to extend the power of the platform.
- Data Models: allow developers to abstract away the search language syntax, making Splunk queries (and thus, functionality) more manageable and portable/shareable.
- Modular Inputs: allow developers to extend Splunk to programmatically manage custom data input functionality via REST.