This document provides an overview of Splunk's developer platform. It introduces Jon Rooney, Director of Developer Marketing at Splunk, and Damien Dallimore, Developer Evangelist. It discusses how Splunk can help with application development challenges like visibility across the development lifecycle. It also demonstrates how Splunk can integrate with the development process using tools like its REST API and SDKs. The document highlights Splunk's modular inputs, web framework, and opportunities for custom visualizations and search commands. Overall, it aims to showcase Splunk's powerful platform for developers.
Splunk for JMX App overview (configuration, deployment, tips and tricks). Developing JMX logic in your application. Splunking other JVM logs and profiling traces. The JVM application landscape and why it's such a rich source of Splunkable machine data. Developing new Splunkbase apps to leverage Splunk for JMX.
Kubernetes Forum Seoul 2019: Re-architecting Data Platform with KubernetesSeungYong Oh
Session Video: https://youtu.be/7MPH1mknIxE
In this talk, we share Devsisters' journey of migrating its internal data platform including Spark to Kubernetes, with its benefits and issues.
데브시스터즈에서 데이터플랫폼 컴포넌트를 쿠버네티스로 옮기면서 얻은 장점들과 이슈들에 대해 공유합니다.
Conference session page:
- English: https://sched.co/WIRK
- Korean: https://sched.co/WYRc
During this brief walkthrough of the setup, configuration and use of the toolset we will show you how to find the trees from the forest in today's modern cloud environments and beyond.
Splunk for JMX App overview (configuration, deployment, tips and tricks). Developing JMX logic in your application. Splunking other JVM logs and profiling traces. The JVM application landscape and why it's such a rich source of Splunkable machine data. Developing new Splunkbase apps to leverage Splunk for JMX.
Kubernetes Forum Seoul 2019: Re-architecting Data Platform with KubernetesSeungYong Oh
Session Video: https://youtu.be/7MPH1mknIxE
In this talk, we share Devsisters' journey of migrating its internal data platform including Spark to Kubernetes, with its benefits and issues.
데브시스터즈에서 데이터플랫폼 컴포넌트를 쿠버네티스로 옮기면서 얻은 장점들과 이슈들에 대해 공유합니다.
Conference session page:
- English: https://sched.co/WIRK
- Korean: https://sched.co/WYRc
During this brief walkthrough of the setup, configuration and use of the toolset we will show you how to find the trees from the forest in today's modern cloud environments and beyond.
Developing Real-Time Data Pipelines with Apache KafkaJoe Stein
Developing Real-Time Data Pipelines with Apache Kafka http://kafka.apache.org/ is an introduction for developers about why and how to use Apache Kafka. Apache Kafka is a publish-subscribe messaging system rethought of as a distributed commit log. Kafka is designed to allow a single cluster to serve as the central data backbone. A single Kafka broker can handle hundreds of megabytes of reads and writes per second from thousands of clients. It can be elastically and transparently expanded without downtime. Data streams are partitioned and spread over a cluster of machines to allow data streams larger than the capability of any single machine and to allow clusters of coordinated consumers. Messages are persisted on disk and replicated within the cluster to prevent data loss. Each broker can handle terabytes of messages. For the Spring user, Spring Integration Kafka and Spring XD provide integration with Apache Kafka.
Apache Kafka lies at the heart of the largest data pipelines, handling trillions of messages and petabytes of data every day. Learn the right approach for getting the most out of Kafka from the experts at LinkedIn and Confluent. Todd Palino and Gwen Shapira demonstrate how to monitor, optimize, and troubleshoot performance of your data pipelines—from producer to consumer, development to production—as they explore some of the common problems that Kafka developers and administrators encounter when they take Apache Kafka from a proof of concept to production usage. Too often, systems are overprovisioned and underutilized and still have trouble meeting reasonable performance agreements.
Topics include:
- What latencies and throughputs you should expect from Kafka
- How to select hardware and size components
- What you should be monitoring
- Design patterns and antipatterns for client applications
- How to go about diagnosing performance bottlenecks
- Which configurations to examine and which ones to avoid
Monitoring Docker at Scale - Docker San Francisco Meetup - August 11, 2015Datadog
In this session I showed building a multi-container app from beginning to end, using Docker, Docker-Machine, Docker-Compose and everything in between. You can even try it out yourself using the link in the deck to a repo on GitHub.
AWS는 클라우드 기반의 기계 학습 및 딥러닝 기술을 제공하는 인공 지능 서비스 개발 플랫폼을 제공합니다. AWS Deep Learning AMI를 사용하면 심도 깊은 학습을 실행할 수 있습니다. 정교한 맞춤형 AI 모델을 개발하며, 새로운 알고리즘을 실험하기 위한 오픈 소스 심층 학습 엔진(Apache MXNet 등) AMI를 GPU 기반 인스턴스와 클러스터를 스팟 인스턴스를 통해 비용 효율적으로 구성하여 운영하는 방법을 안내합니다.
With more than 140 million users, KakaoTalk is the most popular mobile messaging platform in South Korea. The team at daumkakao has been using OpenStack with the intention for tranforming the current legacy infrastructure into scale out based cloud to build and offer new services for its users. In this session, we'd like to share our experiences with the OpenStack community, specifically in regards to meeting our needs for networking with Neutron.OpenStack Neutron offers a lot of methods to implement networking for VMs and containers. For production operations, VM migration can be a common activity to manage resources and improve uptime. It's not hard using shared storage like Ceph, but network settings, such as IP addresses need to be preserved. With a shared storage environment, an image can be attached anywhere inside of a data center, but a service IP for a virtual machine is different story. And when you don't use the floating IPs, keeping the same IP across a data center-wide set of VLANs is hard job.To maintain a virtual machine's IP settings and balance IPs between VLANS, we tried several options including overlay, SDN, and NFV technologies. In the end we came to use a route-only network for our virtual machine networks, leveraging technology like Quagga for RIP, OSPF BGP integrated with Neutron.
Come può .NET contribuire alla Data Science? Cosa è .NET Interactive? Cosa c'entrano i notebook? E Apache Spark? E il pythonismo? E Azure? Vediamo in questa sessione di mettere in ordine le idee.
Pulsar Architectural Patterns for CI/CD Automation and Self-Service_Devin BostStreamNative
We examine real-world architectural patterns involving Apache Pulsar to automate the creation of function and pub/sub flows for improved operational scalability and ease of management. We’ll cover CI/CD automation patterns and reveal our innovative approach of leveraging streaming data to create a self-service platform that automates the provisioning of new users. We will also demonstrate the innovative approach of creating function flows through patterns and configuration, enabling non-developer users to create entire function flows simply by changing configurations. These patterns enable us to drive the automation of managing Pulsar to a whole new level. We also cover CI/CD for on-prem, GCP, and AWS users.
This is Part 2 of this presentation: https://www.youtube.com/watch?v=pmaCG...
In summary, we will cover:
CI/CD for on-prem, GCP, and AWS users
Automated creation of function flows by configuration
Automated provisioning of pub/sub users and topics
Architectural patterns and best practices that enable automation
Overstock has leveraged Pulsar as the backbone of a self-service data fabric, a unified data platform to enable users to publish and consume data across the company and integrate with other services. We utilized Pulsar to solve a data governance problem, and Pulsar has performed marvelously. To support our real-world production use cases, we have developed message flows, integrations, and architectural patterns to solve common use cases, maximize value, simplify ease-of-use, automate management, and unify company data and services around this new platform.
Big data event streaming is very common part of any big data Architecture. Of the available open source big data streaming technologies Apache Kafka stands out because of it realtime, distributed, and reliable characteristics. This is possible because of the Kafka Architecture. This talk highlights those features.
Infrastructure-as-code: bridging the gap between Devs and OpsMykyta Protsenko
Ops are overwhelmed with support. Devs are mad because their cannot deploy the changes as fast as they want. Sounds familiar?
Infrastructure-as-code can make your life easier by empowering developers and reducing operations' routine toil. It can cut down the lead time for infrastructure provisioning from hours or even days to minutes.
This talk reviews several IaC tools and approaches, showing how to integrate them into continuous delivery pipeline. It covers the problems and challenges that engineers may face while working with infrastructure-as-code tools and provides a few hands-on recipes to address them.
Python is a great programming language that works great with Cassandra. If your goal is to get your project into production quickly and iterate fast, Python is a great solution.
These slides are an introduction to the hands on portion from GitHub. https://github.com/rustyrazorblade/python-presentation
Putting Kafka In Jail – Best Practices To Run Kafka On Kubernetes & DC/OSLightbend
Apache Kafka–part of Lightbend Fast Data Platform–is a distributed streaming platform that is best suited to run close to the metal on dedicated machines in statically defined clusters. For most enterprises, however, these fixed clusters are quickly becoming extinct in favor of mixed-use clusters that take advantage of all infrastructure resources available.
In this webinar by Sean Glover, Fast Data Engineer at Lightbend, we will review leading Kafka implementations on DC/OS and Kubernetes to see how they reliably run Kafka in container orchestrated clusters and reduce the overhead for a number of common operational tasks with standard cluster resource manager features. You will learn specifically about concerns like:
* The need for greater operational knowhow to do common tasks with Kafka in static clusters, such as applying broker configuration updates, upgrading to a new version, and adding or decommissioning brokers.
* The best way to provide resources to stateful technologies while in a mixed-use cluster, noting the importance of disk space as one of Kafka’s most important resource requirements.
* How to address the particular needs of stateful services in a model that natively favors stateless, transient services.
Lifting the Blinds: Monitoring Windows Server 2012Datadog
Operating systems monitor resources continuously in order to effectively schedule processes.
In this webinar, Evan Mouzakitis (Datadog) discusses how to get operational data from Windows Server 2012 using a variety of native tools.
Developing Real-Time Data Pipelines with Apache KafkaJoe Stein
Developing Real-Time Data Pipelines with Apache Kafka http://kafka.apache.org/ is an introduction for developers about why and how to use Apache Kafka. Apache Kafka is a publish-subscribe messaging system rethought of as a distributed commit log. Kafka is designed to allow a single cluster to serve as the central data backbone. A single Kafka broker can handle hundreds of megabytes of reads and writes per second from thousands of clients. It can be elastically and transparently expanded without downtime. Data streams are partitioned and spread over a cluster of machines to allow data streams larger than the capability of any single machine and to allow clusters of coordinated consumers. Messages are persisted on disk and replicated within the cluster to prevent data loss. Each broker can handle terabytes of messages. For the Spring user, Spring Integration Kafka and Spring XD provide integration with Apache Kafka.
Apache Kafka lies at the heart of the largest data pipelines, handling trillions of messages and petabytes of data every day. Learn the right approach for getting the most out of Kafka from the experts at LinkedIn and Confluent. Todd Palino and Gwen Shapira demonstrate how to monitor, optimize, and troubleshoot performance of your data pipelines—from producer to consumer, development to production—as they explore some of the common problems that Kafka developers and administrators encounter when they take Apache Kafka from a proof of concept to production usage. Too often, systems are overprovisioned and underutilized and still have trouble meeting reasonable performance agreements.
Topics include:
- What latencies and throughputs you should expect from Kafka
- How to select hardware and size components
- What you should be monitoring
- Design patterns and antipatterns for client applications
- How to go about diagnosing performance bottlenecks
- Which configurations to examine and which ones to avoid
Monitoring Docker at Scale - Docker San Francisco Meetup - August 11, 2015Datadog
In this session I showed building a multi-container app from beginning to end, using Docker, Docker-Machine, Docker-Compose and everything in between. You can even try it out yourself using the link in the deck to a repo on GitHub.
AWS는 클라우드 기반의 기계 학습 및 딥러닝 기술을 제공하는 인공 지능 서비스 개발 플랫폼을 제공합니다. AWS Deep Learning AMI를 사용하면 심도 깊은 학습을 실행할 수 있습니다. 정교한 맞춤형 AI 모델을 개발하며, 새로운 알고리즘을 실험하기 위한 오픈 소스 심층 학습 엔진(Apache MXNet 등) AMI를 GPU 기반 인스턴스와 클러스터를 스팟 인스턴스를 통해 비용 효율적으로 구성하여 운영하는 방법을 안내합니다.
With more than 140 million users, KakaoTalk is the most popular mobile messaging platform in South Korea. The team at daumkakao has been using OpenStack with the intention for tranforming the current legacy infrastructure into scale out based cloud to build and offer new services for its users. In this session, we'd like to share our experiences with the OpenStack community, specifically in regards to meeting our needs for networking with Neutron.OpenStack Neutron offers a lot of methods to implement networking for VMs and containers. For production operations, VM migration can be a common activity to manage resources and improve uptime. It's not hard using shared storage like Ceph, but network settings, such as IP addresses need to be preserved. With a shared storage environment, an image can be attached anywhere inside of a data center, but a service IP for a virtual machine is different story. And when you don't use the floating IPs, keeping the same IP across a data center-wide set of VLANs is hard job.To maintain a virtual machine's IP settings and balance IPs between VLANS, we tried several options including overlay, SDN, and NFV technologies. In the end we came to use a route-only network for our virtual machine networks, leveraging technology like Quagga for RIP, OSPF BGP integrated with Neutron.
Come può .NET contribuire alla Data Science? Cosa è .NET Interactive? Cosa c'entrano i notebook? E Apache Spark? E il pythonismo? E Azure? Vediamo in questa sessione di mettere in ordine le idee.
Pulsar Architectural Patterns for CI/CD Automation and Self-Service_Devin BostStreamNative
We examine real-world architectural patterns involving Apache Pulsar to automate the creation of function and pub/sub flows for improved operational scalability and ease of management. We’ll cover CI/CD automation patterns and reveal our innovative approach of leveraging streaming data to create a self-service platform that automates the provisioning of new users. We will also demonstrate the innovative approach of creating function flows through patterns and configuration, enabling non-developer users to create entire function flows simply by changing configurations. These patterns enable us to drive the automation of managing Pulsar to a whole new level. We also cover CI/CD for on-prem, GCP, and AWS users.
This is Part 2 of this presentation: https://www.youtube.com/watch?v=pmaCG...
In summary, we will cover:
CI/CD for on-prem, GCP, and AWS users
Automated creation of function flows by configuration
Automated provisioning of pub/sub users and topics
Architectural patterns and best practices that enable automation
Overstock has leveraged Pulsar as the backbone of a self-service data fabric, a unified data platform to enable users to publish and consume data across the company and integrate with other services. We utilized Pulsar to solve a data governance problem, and Pulsar has performed marvelously. To support our real-world production use cases, we have developed message flows, integrations, and architectural patterns to solve common use cases, maximize value, simplify ease-of-use, automate management, and unify company data and services around this new platform.
Big data event streaming is very common part of any big data Architecture. Of the available open source big data streaming technologies Apache Kafka stands out because of it realtime, distributed, and reliable characteristics. This is possible because of the Kafka Architecture. This talk highlights those features.
Infrastructure-as-code: bridging the gap between Devs and OpsMykyta Protsenko
Ops are overwhelmed with support. Devs are mad because their cannot deploy the changes as fast as they want. Sounds familiar?
Infrastructure-as-code can make your life easier by empowering developers and reducing operations' routine toil. It can cut down the lead time for infrastructure provisioning from hours or even days to minutes.
This talk reviews several IaC tools and approaches, showing how to integrate them into continuous delivery pipeline. It covers the problems and challenges that engineers may face while working with infrastructure-as-code tools and provides a few hands-on recipes to address them.
Python is a great programming language that works great with Cassandra. If your goal is to get your project into production quickly and iterate fast, Python is a great solution.
These slides are an introduction to the hands on portion from GitHub. https://github.com/rustyrazorblade/python-presentation
Putting Kafka In Jail – Best Practices To Run Kafka On Kubernetes & DC/OSLightbend
Apache Kafka–part of Lightbend Fast Data Platform–is a distributed streaming platform that is best suited to run close to the metal on dedicated machines in statically defined clusters. For most enterprises, however, these fixed clusters are quickly becoming extinct in favor of mixed-use clusters that take advantage of all infrastructure resources available.
In this webinar by Sean Glover, Fast Data Engineer at Lightbend, we will review leading Kafka implementations on DC/OS and Kubernetes to see how they reliably run Kafka in container orchestrated clusters and reduce the overhead for a number of common operational tasks with standard cluster resource manager features. You will learn specifically about concerns like:
* The need for greater operational knowhow to do common tasks with Kafka in static clusters, such as applying broker configuration updates, upgrading to a new version, and adding or decommissioning brokers.
* The best way to provide resources to stateful technologies while in a mixed-use cluster, noting the importance of disk space as one of Kafka’s most important resource requirements.
* How to address the particular needs of stateful services in a model that natively favors stateless, transient services.
Lifting the Blinds: Monitoring Windows Server 2012Datadog
Operating systems monitor resources continuously in order to effectively schedule processes.
In this webinar, Evan Mouzakitis (Datadog) discusses how to get operational data from Windows Server 2012 using a variety of native tools.
Integrating Splunk into your Spring ApplicationsDamien Dallimore
How much visibility do you really have into your Spring applications? How effectively are you capturing,harnessing and correlating the logs, metrics, & messages from your Spring applications that can be used to deliver this visibility ? What tools and techniques are you providing your Spring developers with to better create and utilize this mass of machine data ? In this session I'll answer these questions and show how Splunk can be used to not only provide historical and realtime visibility into your Spring applications , but also as a platform that developers can use to become more "devops effective" & easily create custom big data integrations and standalone solutions.I'll discuss and demonstrate many of Splunk's Java apps,frameworks and SDK and also cover the Spring Integration Adaptors for Splunk.
Get Devops Training in Chennai with real-time experts at Besant Technologies, OMR. We believe that learning Devops with practical and theoretical will be the easiest way to understand the technology in quick manner. We designed this Devops from basic level to the latest advanced level
http://www.traininginsholinganallur.in/devops-training-in-chennai.html
Webinar: Site Search in an Hour with FusionLucidworks
Using Lucidworks View and Fusion 3, you can easily build and deploy site search in less than one hour. Even with multiple data sources, data transformations, and user interface development, a full enterprise search project can be completed in just an hour compared to the usual 6 months.
Sergii Bielskyi "Azure Logic App and building modern cloud native apps"Fwdays
I would like to share my experience of using Azure Logic App as a tool to build cloud native applications as easy and fast as we can. I will explain how we can use SDKs for our needs and how to use the cloud interface to simplify the process. During the demo, I will solve the problem that often happens especially with me when I am not at home.
Well, the story will be about IoT device and how we can user Logic app to notify me when electricity is turned off. Also, I will demonstrate how we can use computer vision into the Logic App.
Join Lucidworks Senior Director of Product Avi Raju and Technical Engagement Manager Andy Oliver for a guided tour of what’s new and improved with our latest release of Fusion 4.1, including:
- App Studio integration so you can go from data ingest to a working search application in minutes.
- Fusion Apps, a grouping of objects that can be exported and shared amongst Fusion instances, reducing time to deployment for new applications.
- New data acquisition capabilities to load and analyze massive amounts of data from data stores like Cassandra, Hive, and HBase.
- An improved Connectors SDK that allows data to be ingested from any data source.
- Improved SQL capabilities to query your index with commands and tools you already know with subsecond (~250ms) response time across billions of documents including endpoints to connect with popular BI tools like Tableau, Microsoft Power BI, and Apache Zeppelin.
Docs as Part of the Product - Open Source Summit North America 2018Den Delimarsky
The presentation showcased at the Open Source Summit North America 2018 in Vancouver, BC. It covers the learnings from transitioning the MSDN site functionality and content to docs.microsoft.com.
Splunk SDKs make it faster and more efficient to program using the Splunk REST API using constructs and syntax familiar to developers who are experienced in Java, Python, JavaScript and PHP. This makes it easier to integrate data from Splunk with other applications across the enterprise.
Similar to SplunkLive London 2014 Developer Presentation (20)
GAINING APPLICATION LIFECYCLE INTELLIGENCE
Applied Spring Track
Today we are facing an ever-increasing speed of product delivery. DevOps practices
like continuous integration and deployment increase the dependence of systems
like task tracking and source code repositories with build servers and test suites.
With data moving rapidly through these different tools, it becomes challenging to
maintain a grasp of the process, especially as the data is distributed and in a variety
of formats. But it is still critical to maintain full visibility of the product development
journey – from user stories to production data. By starting at the beginning of the
Product Development Lifecycle, you can track a problem in production all the way
back to the code that was checked into the build and the developer responsible for
the code.
In this session I'll demonstrate some of the ways in which Splunk software can be
used to collect and correlate data throughout the various stages of the lifecycle of
your code, to ultimately make you more efficient and make your code better.
Levelwise PageRank with Loop-Based Dead End Handling Strategy : SHORT REPORT ...Subhajit Sahu
Abstract — Levelwise PageRank is an alternative method of PageRank computation which decomposes the input graph into a directed acyclic block-graph of strongly connected components, and processes them in topological order, one level at a time. This enables calculation for ranks in a distributed fashion without per-iteration communication, unlike the standard method where all vertices are processed in each iteration. It however comes with a precondition of the absence of dead ends in the input graph. Here, the native non-distributed performance of Levelwise PageRank was compared against Monolithic PageRank on a CPU as well as a GPU. To ensure a fair comparison, Monolithic PageRank was also performed on a graph where vertices were split by components. Results indicate that Levelwise PageRank is about as fast as Monolithic PageRank on the CPU, but quite a bit slower on the GPU. Slowdown on the GPU is likely caused by a large submission of small workloads, and expected to be non-issue when the computation is performed on massive graphs.
Adjusting primitives for graph : SHORT REPORT / NOTESSubhajit Sahu
Graph algorithms, like PageRank Compressed Sparse Row (CSR) is an adjacency-list based graph representation that is
Multiply with different modes (map)
1. Performance of sequential execution based vs OpenMP based vector multiply.
2. Comparing various launch configs for CUDA based vector multiply.
Sum with different storage types (reduce)
1. Performance of vector element sum using float vs bfloat16 as the storage type.
Sum with different modes (reduce)
1. Performance of sequential execution based vs OpenMP based vector element sum.
2. Performance of memcpy vs in-place based CUDA based vector element sum.
3. Comparing various launch configs for CUDA based vector element sum (memcpy).
4. Comparing various launch configs for CUDA based vector element sum (in-place).
Sum with in-place strategies of CUDA mode (reduce)
1. Comparing various launch configs for CUDA based vector element sum (in-place).
2. Who is Jon Rooney?
Director of Developer Marketing
Splunker since April 2012
Based in Splunk’s Seattle Office
Veteran of Microsoft, start-ups,
Accenture
2
4. Application Development Challenges
4
Build Deployment
Server
QA /
Test
Staging
Environment
Continuous
Integration /
Build Server
Source
Repository
Task Tracking
Local Build
5. Application Development Challenges
5
Build Deployment
Server
QA /
Test
Staging
Environment
Continuous
Integration /
Build Server
Source
Repository
Task Tracking
Local Build
Lack of visibility across the product
development lifecycle
Slows down the ability to detect and
troubleshoot errors
Limited visibility into application
usage and performance
6. 6
CI / Build
Servers
Project and Issue
Tracking
Code
Repository
QA / Testing
Tools
Splunk can solve these problems
Deployment
Servers
7. Splunk for Application Lifecycle Intelligence
7
Reduce Time to
Market
Resolve issues faster
Gain Agility
Improve Code Quality
and Lifecycle Visibility
Generate
Application Insights
Instrument
Analytics
11. Evolving the Splunk Platform
Collection
Indexing
Search Processing Language
Core Functions
Inputs, Apps, Other
Content
SDKs
Operational Intelligence Platform
Content
Core Engine
User and Developer Interfaces
Web Framework
REST API
12. Powerful Platform for Enterprise Developers
12
REST API
Build Splunk Apps Extend and Integrate Splunk
Simple XML
JavaScript
Django
Web
Framework
Java
JavaScript
Python
Ruby
C#
PHP
Data Models
Search Extensibility
Modular Inputs
SDKs
13. Splunk Web Framework
13
Familiar Technologies
- Code with JavaScript & Django/Python
- HTML5/CSS/JS Support
- Built on JQuery & Backbone.js
Flexible and Extensible
- Create custom layouts
- Integrate visualizations like Sankey
charts, heat maps and bubble charts
- SimpleXML to HTML Conversion
Simple XML
JavaScript
Django
Web
Framework
14. Log directly to
Splunk via TCP,
UDP, HTTP
Integrate search
results with other
applications using
custom
visualizations
Create and run
searches from
other applications
The REST API and SDKs
14
VisualizeSearch Manage
Add/Delete Users
Manage Inputs
Index
16. Who is Damien Dallimore
16
Worldwide Developer Evangelist @ Splunk
I develop
I talk about developing
Helping to build the Splunk developer ecosystem
Came from the Splunk Community
Once was a customer
Coder
17. I develop
17
All 100% free and open sourced , published to Splunk Apps , source code on Github
18. An Open Platform for Developers
18
Splunk is an open and extensible platform at numerous different
touchpoints for developers
Extensibility creates ecosystems
19. A Developer’s Smorgasboard
19
• CLI
• REST API
• 6 language SDKs
• Splunkbase Apps / Add-ons
• Custom search commands
• Scripted Inputs
• Modular Inputs
• The Web Framework
• Standard HTML/Javascript/CSS
• Data Models
• External scripted lookups
• Custom REST Endpoints
• Tools , utilitys and librarys
• Integrations with other software
frameworks (Spring)
• Hadoop dev with HUNK
• Custom user interfaces /
visualizations
• Mobile with Bugsense
• Custom data connectors (ODBC)
• Custom authentication handlers
20. Which Splunk Product for Devs ?
20
Splunk> Enterprise : Free to download and use. Index 500 MB/day.
Splunk> Cloud : Premium, cloud hosted. Full Enterprise stack.
Splunk> AMIs : BYOL versions for Amazon AWS Cloud.
Hunk> : Splunk for data in Hadoop clusters. Same platform, same easy apps.
Splunk> Storm : Free to use, cloud hosted. 20GB/30days.
22. Modular Inputs
22
• Extend the Splunk framework to define a custom input capability, just like the standard inputs you are
familiar with (TCP/UDP/File etc…)
• Splunk treats your custom input definitions as if they were part of Splunk's native inputs, totally
integrated first class citizen objects in Splunk
• Users interactively create and update your custom inputs using Splunk manager, just as they do for
native inputs.
• When deploying without a UI , you push out the inputs.conf file.
• All the properties are fully manageable via the REST API
• Version 5.0 +
24. Developing
24
• My preference is to use Python, however any language can be used.
• http://docs.splunk.com/Documentation/Splunk/latest/AdvancedDev/ModInputsIntro
• There is a certain amount of “plumbing” to put in place , so I like utilities that take care of this for
you, so you can just focus on the business logic.
• Java,Python,C# SDKs also have Modular Input APIs
• Eclipse plugin has a wizard for creating Modular Inputs in Java
25. REST : The Data Potential
25
• Twitter
• Foursquare
• LinkedIn
• Facebook
• Fitbit
• Amazon
• Yahoo
• Reddit
• YouTube
• Flickr
• Wikipedia
• GNIP
• Box
• Okta
• Datasift
• Google APIs
• Weather Services
• Seismic monitoring
• Publicly available socio-economic data
• Traffic data
• Stock monitoring
• Security service providers
• Proprietary systems and platforms
• Other “data related” software products
• The REST “dataverse” is vast , but I think you get
the point.
There is a world of data out there available via REST that can be brought into Splunk, correlated and
enriched against your existing data, or used for entirely new uses cases that you might conceive of once you
see what is available and where your data might take you.
26. You are only limited by your own “data imagination”
26
29. Splunk Search
29
• Splunk’s search and querying language is called SPL
• Allows you to search, analyze and manipulate your data.
• Designed with the “unix pipeline” in mind – a “search pipeline”
• From a (simplistic) mental point of view:
– Series of commands
– Each commands takes the input of the previous one
– Each command outputs a sparse table
30. Splunk Search - Example 1:
30
“index=_internal foo | eval bar=... | stats count by bar”
• Initial command fetches result from index/raw data store, and outputs a
table, with a row for each event, and a column for each field (not all rows
have all columns)
• Second command adds a new column to each row, “bar”
• Third command looks at all the unique values of “bar”, and counts how
many rows each value has.
• Note that the last command completely transformed the table, hence it is a
“transforming” or “non-streaming” command.
• The second command was merely additive, known as a “streaming”
command.
• You can also have “generating” commands ie: inputlookup
31. Custom Search Commands
31
• Just like the “eval” or “stats” commands, you can write your own
“search commands”.
• Python scripts which take data on stdin, and output data on stdout
• Data comes in/out as CSV (with special handling for MV fields)
• Many of Splunk’s builtin commands are written as Python scripts
(e.g. head, return, transpose)
32. Custom Search Commands: Building Blocks
32
Custom search commands are built in two parts:
• A Python script containing the implementation
• An entry in commands.conf declaring configuring the command
• Splunk SDK for Python has librarys and examples for creating custom
search commands
35. Splunk Web Framework
35
Familiar Technologies
- Code with JavaScript & Django/Python
- HTML5/CSS/JS Support
- Built on JQuery & Backbone.js
Flexible and Extensible
- Create custom layouts
- Integrate visualizations like Sankey
charts, heat maps and bubble charts
- SimpleXML to HTML Conversion
Simple XML
JavaScript
Django
Web
Framework
36. Splunk JS Stack & Django Bindings Concepts
36
Managers
• Search Query Wrappers
• SearchManager and
SaveSearch Manager
• Include Search Parameters
• Available within Django or
JavaScript
37. Splunk JS Stack & Django Bindings Concepts
37
Splunk Views
• UI widgets
• Designed to work with Search
Managers
• Charts, Maps, Inputs, Timelin
e, TimePicker, etc.
38. Splunk JS Stack & Django Bindings Concepts
38
URL Maps & Django View
• URL Maps enable custom
routes
• Django Views, enable custom
logic to provide to templates
39. Splunk JS Stack & Django Bindings Concepts
39
Templates & Template Tags
• Templates enable quick
layout options
• Templates support
inheritance of other
templates
• Tags call a Python function
can be used for:
– Text manipulation
– Flow control
– Load external information
– … and more
40. Splunk JS Stack & Django Bindings Concepts
40
Data Binding using Tokens
• Token based data-binding
mechanism that keep shared
data in sync.
• Enables in-page interactivity
41. Web Framework Toolkit App
41
• Improved productivity for developing your own
Web Framework based apps
• Reusable Visualization
• Packaged to lower the effort of adding cool
visualizations
• Improved Developer focused Command Line
Tools
• Getting started templates
• Automates common tasks
• Includes examples pages demonstrating
advanced concepts.
42. SimpleXML converted to HTML
42
• Transition a Simple XML page to HTML/JS
• Option to overwrite/edit existing dashboard or create new
• Full Splunk JS Stack is available (Add custom viz, tokens, etc.)
• Does not include Django backend
• No support for visual dashboard editor or PDF printing
50. My Guiding Viz Principle
50
The visualization must be simple and intuitive to understand and derive meaning from at a glance.
Cool viz , but what are you telling me ?
51. So many options , which one for me ?
Splunk Web Form Editor
– If you are not a coder , not familiar with Simple XML
Edit Simple XML
– Familiar with Simple XML , what to customize more
Convert Simple XML to HTML/JS
– Coder , want to see the underlying JS/HTML , want custom UI behavior above Simple
XML, want to use some other JS/CSS
Simple XML JS Import
– Closely related to the above , perhaps you still want dashboard editing / PDF export
Django
– Previous benefits + want to leverage Django tags , want custom server side processing in
Django views
51
54. Company Overview
The right toolset for analyzing and troubleshooting mobile apps
in real-time
54
Deliver high quality, engaging apps.
Splunk BugSense
55. 55
Capabilities
Mobile Data
HTML5 dashboard
• Actionable reports
• Easy to use
How It Works
Cross Platform SDKs
• Install in < 5 min
• Crashes/Errors
• Events
• Sessions
• Transactions
Bug Sense Cloud
• Highly Scalable
• Secure
• Cloud Service
• Highly available
• Cost-effective
56. Integration (Android)
1. Add the .jar (download or use Maven/Gradle) & import
2. Add the permissions
3. Initialize:
61. WHY should you develop ?
61
Make money , Promote your company, Make sales !
Community and collaboration
Share / Give Back
Get a job / Build a career
Learn new skills / Educate yourself and others
Hadoop productivity
Do good
Open up new data sources for others to collaborate on
We talk a lot about the how , what , where and who ….. but what about the WHY
65. Where to go for More Info
• Tutorials, Code Samples, Getting Started, Downloads
– http://dev.splunk.com/
• Splunk Apps
– https://apps.splunk.com
• GitHub
– https://github.com/splunk/
• Twitter
– https://twitter.com/splunkdev
• Blogs
– http://blogs.splunk.com/dev/
65
66. The 5th Annual Splunk WWUC
• 50+ Customer speakers
• 30+ Apps in Splunk Apps
Showcase
• 30+ Technology Partners
• Ask The Experts
• Sales Meetings
• Business Value ROI booth
conf.splunk.com
• Las Vegas: Oct 6-9, 2014
• The MGM Grand Hotel
• 4000+ IT and Business Professionals
• 3 days of content, 130+ sessions
• 3 days of Splunk University
• Get Certified!
While Splunk is well-known as a world-class solution for IT and security professionals around the world, the Splunk platform
With the increased speed of product delivery – from an annual cadence with packaged software to continuous deployment with modern cloud services – organizations are under increasing pressure to run a well-oiled, fault tolerant, rapid delivery pipeline in their product development lifecycle. DevOps practices like continuous integration, automated configuration and continuous deployment increase the dependence of systems like task tracking and source code repositories with build servers and test suites.
With data moving rapidly through these different tools, it becomes challenging to maintain a grasp of the process. Despite challenges with silo-ed data in various tools, it is still critical to maintain full visibility of the product development journey – from user stories to production data. The various systems and tools contain the data necessary – like task data stored in JIRA, acceptance criteria and test results and the status of builds in QA and staging environments – is trapped in different formats in various tools, without an easily, flexible way to correlate and gain insights from.
Splunk increases the speed and efficiency of application development and testing, reducing time to market and enabling DevOps agility with connected visibility across the lifecycle. By starting at the beginning of the Product Development Lifecycle, you can track a problem in production all the way back to the code that was checked into the build and the developer responsible for the code. Using Splunk throughout the Product Development lifecycle allows the customers to be more agile and reduce time to market. Splunk enables continuous development and delivery of enterprise applications across the entire lifecycle, making the entire continuous delivery process seamless and frictionless for developers, testers, and operations personnel. Splunk deliver Application Lifecycle Intelligence – real time, mission critical visibility into every step, system and process involved in shipping new product to your customers.
Resolve Issues FasterSplunk enables developers to quickly trace errors to reduce time to market. As code moves from unit testing to staging environments, Splunk can be invaluable in tracking and benchmarking high-volume stress tests.Improve Code Quality and Lifecycle VisibilitySplunk provides visibility into events and activities in the disparate tools and systems that drive the development lifecycle, from task management and code repositories to build and deployment servers. Get real-time insights into automated processes like builds and tests to quickly identify errors to support rapid release cycles.Usage analyticsSplunk's late binding schema and powerful search language help developers analyze semantic data and deliver powerful business-level insights around usertrends, preferences, feature adoption, and more. By writing intelligence in the application logs through the practice of semantic logging, developers can use thelogs generated from custom applications to get powerful business insights without building or implementing special-purpose software. You have no controlover other systems events.With custom application development, you have full control over events that you write. While most events are written by developersto help them debug and some events are written to form an audit trail, semantic events are written explicitly for the gathering of analytics
We have many customers who have applied Splunk’s ability to collect, index and analyze disparate machine data in real time to the product development process, gaining critical Application Lifecycle intelligence.
As the Splunk platform evolves you can see how the Development capabilities have expanded to enable developers to better build on Splunk’s core capabilities. Let’s double click into the Developer platform….
BUILD SPLUNK APPSThe Splunk Web Framework makes building a Splunk app looks and feels like building any modern web application. The Simple Dashboard Editor makes it easy to BUILD interactive dashboards and user workflows as well as add custom styling, behavior and visualizations. Simple XML is ideal for fast, lightweight app customization and building. Simple XML development requires minimal coding knowledge and is well-suited for Splunk power users in IT to get fast visualization and analytics from their machine data. Simple XML also lets the developer “escape” to HTML with one click to do more powerful customization and integration with JavaScript. Developers looking for more advanced functionality and capabilities can build Splunk apps from the ground up using popular, standards-based web technologies: JavaScript and Django. The Splunk Web Framework lets developers quickly create Splunk apps by using prebuilt components, styles, templates, and reusable samples as well as supporting the development of custom logic, interactions, components, and UI. Developers can choose to program their Splunk app using Simple XML, JavaScript or Django (or any combination thereof).EXTEND AND INTEGRATE SPLUNKSplunk Enterprise is a robust, fully-integrated platform that enables developers to INTEGRATE data and functionality from Splunk software into applications across the organization using Software Development Kits (SDKs) for Java, JavaScript, C#, Python, PHP and Ruby. These SDKs make it easier to code to the open REST API that sits on top of the Splunk Engine. With almost 200 endpoints, the REST API lets developers do programmatically what any end user can do in the UI and more. The Splunk SDKs include documentation, code samples, resources and tools to make it faster and more efficient to program against the Splunk REST API using constructs and syntax familiar to developers experienced with Java, Python, JavaScript, PHP, Ruby and C#. Developers can easily manage HTTP access, authentication and namespaces in just a few lines of code. Developers can use the Splunk SDKs to: - Run real-time searches and retrieve Splunk data from line-of-business systems like Customer Service applications - Integrate data and visualizations (charts, tables) from Splunk into BI tools and reporting dashboards- Build mobile applications with real-time KPI dashboards and alerts powered by Splunk - Log directly to Splunk from remote devices and applications via TCP, UDP and HTTP- Build customer-facing dashboards in your applications powered by user-specific data in Splunk - Manage a Splunk instance, including adding and removing users as well as creating data inputs from an application outside of Splunk- Programmatically extract data from Splunk for long-term data warehousingDevelopers can EXTEND the power of Splunk software with programmatic control over search commands, data sources and data enrichment. Splunk Enterprise offers search extensibility through: - Custom Search Commands - developers can add a custom search script (in Python) to Splunk to create own search commands. To build a search that runs recursively, developers need to make calls directly to the REST API- Scripted Lookups: developers can programmatically script lookups via Python.- Scripted Alerts: can trigger a shell script or batch file (we provide guidance for Python and PERL).- Search Macros: make chunks of a search reuseable in multiple places, including saved and ad hoc searches. Splunk also provides developers with other mechanisms to extend the power of the platform.-Data Models: allow developers to abstract away the search language syntax, making Splunk queries (and thus, functionality) more manageable and portable/shareable. - Modular Inputs: allow developers to extend Splunk to programmatically manage custom data input functionality via REST.
Splunk is a fully-integrated platform that delivers rapid time-to-value to application developers. Developers can build robust applications on Splunk that deliver real-time business insights like clickstream analysis, IT early-warning systems, security and fraud protection at enterprise-grade scale using the languages, frameworks and tools that they know and love. Applications built on Splunk can deliver segmented, secure data to customers in any UI, powering your core product or service with real-time operational intelligence making Splunk more valuable and more relevant to more users. The Splunk Web Framework makes building a Splunk app looks and feels like building any modern web application. The Simple Dashboard Editor makes it easy to build interactive dashboards and user workflows as well as add custom styling, behavior and visualizations. Simple XML is ideal for fast, lightweight app customization and building. Simple XML development requires minimal coding knowledge and is well-suited for Splunk power users in IT to get fast visualization and analytics from their machine data. Simple XML also lets the developer “escape” to HTML with one click to do more powerful customization and integration with JavaScript. Developers looking for more advanced functionality and capabilities can build Splunk apps from the ground up using popular, standards-based web technologies: JavaScript and Django. The Splunk Web Framework lets developers quickly create Splunk apps by using prebuilt components, styles, templates, and reusable samples as well as supporting the development of custom logic, interactions, components, and UI. Developers can choose to program their Splunk app using Simple XML, JavaScript or Django (or any combination thereof).Use Simple XML for fast, lightweight dashboard building as well as add custom styling, behavior and visualizationsUse JavaScript for client-side development Use Python and the Django framework for server-side development
The Splunk SDKs empower developers to deliver greater operational agility throughout the enterprise by making it easy to integrate data from Splunk with other applications. Splunk provides a fully-documented and supported REST API with nearly 200 endpoints that let developers programmatically index, search and visualize data in Splunk from any application. Splunk’s SDKs, built on that API, make it easy for developers to integrate data from Splunk with other applications across the enterprise, from custom-built mobile reporting apps to off-the-shelf CRM solutions. Splunk offers SDKs for Python, Java, JavaScript, PHP, Ruby and C#. Developers can use the Splunk SDKs to: Run real-time searches and retrieve Splunk data from line-of-business systems like Customer Service applications Integrate data and visualizations (charts, tables) from Splunk into BI tools and reporting dashboardsBuild mobile applications with real-time KPI dashboards and alerts powered by Splunk Log directly to Splunk from remote devices and applications via TCP, UDP and HTTPBuild customer-facing dashboards in your applications powered by user-specific data in Splunk Manage a Splunk instance, including adding and removing users as well as creating data inputs from an application outside of SplunkProgrammatically extract data from Splunk for long-term data warehousingInteract with data stored in Hadoop using HUNK
Not talking about these , but grab me laterAsk audience for who has developed , who are coders
Journey , get data in , search , visualize
Half the battle is often just getting the data in.Data first , sexy next.
Tesla joke
Refer to Twitter demo / sentiment analysis4SQ , create map viz , show html then segway into customsearch commands with haversine
Sentiment search command on twitter screen
Show code from searchcommands_app
Splunk is a fully-integrated platform that delivers rapid time-to-value to application developers. Developers can build robust applications on Splunk that deliver real-time business insights like clickstream analysis, IT early-warning systems, security and fraud protection at enterprise-grade scale using the languages, frameworks and tools that they know and love. Applications built on Splunk can deliver segmented, secure data to customers in any UI, powering your core product or service with real-time operational intelligence making Splunk more valuable and more relevant to more users. The Splunk Web Framework makes building a Splunk app looks and feels like building any modern web application. The Simple Dashboard Editor makes it easy to build interactive dashboards and user workflows as well as add custom styling, behavior and visualizations. Simple XML is ideal for fast, lightweight app customization and building. Simple XML development requires minimal coding knowledge and is well-suited for Splunk power users in IT to get fast visualization and analytics from their machine data. Simple XML also lets the developer “escape” to HTML with one click to do more powerful customization and integration with JavaScript. Developers looking for more advanced functionality and capabilities can build Splunk apps from the ground up using popular, standards-based web technologies: JavaScript and Django. The Splunk Web Framework lets developers quickly create Splunk apps by using prebuilt components, styles, templates, and reusable samples as well as supporting the development of custom logic, interactions, components, and UI. Developers can choose to program their Splunk app using Simple XML, JavaScript or Django (or any combination thereof).Use Simple XML for fast, lightweight dashboard building as well as add custom styling, behavior and visualizationsUse JavaScript for client-side development Use Python and the Django framework for server-side development
The Web Framework Toolkit ships with support for the following visualization libraries:Sankey ChartGlobeBubble ChartForce DirectedCal HeatParallel CoordinatesParallel SetsSunburst (Doughnut++)
Closely relatedsupport for visual dashboard editor or PDF printing
The Web Framework Toolkit ships with support for the following visualization libraries:Sankey ChartGlobeBubble ChartForce DirectedCal HeatParallel CoordinatesParallel SetsSunburst (Doughnut++)
Build even more compelling appsUniversal drilldownIndividual refreshPan/zoomMulti time range pickersChart overlayAxis label rotationMore from control options (checkboxes ,multi input selections )
Make moneyMake your Splunk App a commercial offering. You can list it on Splunk Apps and get the benefit of that web presence and traffic and then redirect users to your own hosted site where your App is monetized. Generate leads , sell splunk licenses , differentiate from the competition.Promote your companyMany Splunk Apps are for the products of our technology partners. Take advantage of this marketing platform , generate more leads for your business via your Splunk creations , sell more license revenue or generate more service revenue if you are a Splunk reseller/partner.Community and CollaborationBecome part of the Splunk developer ecosystem , collaborate on Splunk Apps and Github , creating exciting new Apps that utilize the power of the Splunk platform to tackle innovative new data solutions.Share / Give BackSplunk Apps has loads of great content , most of it free. So why not give back also if you are using a lot of this content.Get a job / Build a careerI’ve seen a lot of resumes over the years. What makes a technical IT resume stand out from the crowd for me is often your involvement in community and collaborative projects.Learn new skills / Educate yourselfThe techniques involved in building Splunk Apps and Add-ons expose you to a rich landscape of software development and general IT skills. Multiple language development, Web development, APIs, Networking skills , OS and Hardware skills, Maths and Statistics skills etc… Splunk is a fantastic IT learning platform. And what better way to get some practical skills than by building something.Hadoop productivityWith the introduction of HUNK , we have significantly lowered the barrier to get productive results with data locked away in HDFS.So you can build a Splunk App to deliver insights on whatever data you have in HDFS that you are finding difficult to unleash.Do goodWalk in the footsteps of our Splunk for Good initiative and use data and your Splunk App to do some good in this world. There is so much publicly available data that could be used for this purpose i.e.: as 1 such idea you could create an App that uses Social Data and Sentiment Analysis to detect and combat cyberbullying.Open up new data sources for others to collaborate onData can sometimes be hard to get at. A Splunk Add-on can make it easier for others to get at that data so they can start being productive. They may even reuse your Add-on as part of a grander Splunk App.Solve problems , make new discoveriesThe true essence of innovation , creating and discovering new things that have not been done before. Use Splunk to make new discoveries in data and bundle this up in a Splunk App for the community to springboard off.Oh , and one more reason , it’s really fun and rewarding to create things !! Especially with the tremendous resource of data that we have at our disposal , you are only limited by your own imaginations.
.conf2013 war our 3rd annual conferenceHeld in Las Vegas at The Cosmopolitan Hotel in September.Goal here is to make our customers smarter, because smarter customers find new ways to use Splunk and tell their colleagues to use Splunk. Specific conference goals:Help customers answer: Where will your data take you?Empower customers with knowledgeFoster deep, supportive relationships within the Splunk communityGarner rich feedback and input to create a better SplunkReinforce Splunk CommunityEquip Customers and Partners with skills for successCreate channel for sharing best practices—expanding use casesLive, in-person venue for trainingFoundation for everything Splunk--future Users’ Conferences, regional user groups, fueling Splunkbase and Splunk Answers…Successful customers=happy customers=more Splunk sales!