SplunkLive! Developer Session
•
7 likes•5,062 views
This document provides an overview of Splunk's developer platform for building applications and customizing Splunk. It discusses the Splunk web framework, REST API, SDKs for various languages, and sample apps. The web framework allows developing custom UIs using familiar technologies like JavaScript and Django. The REST API exposes all of Splunk's functionality and can be used to integrate Splunk with other applications. SDKs simplify making requests to the REST API from languages like Python, Java, and JavaScript. Sample apps demonstrate how to build custom functionality like monitoring devices and generating mood reports. Support resources for developers include the documentation, support site, GitHub, and Twitter account.
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (9)
Similar to SplunkLive! Developer Session
Similar to SplunkLive! Developer Session (20)
More from Splunk
More from Splunk (20)
Recently uploaded
Recently uploaded (20)
SplunkLive! Developer Session
- 1. Copyright © 2013 Splunk Inc. Splunk for Developers
- 2. Agenda Intro to Splunk for Application Development Splunk Developer Platform Web Framework REST API SDKs and Tools Some Sample Apps Support and Community 2
- 3. Setting expectations • This session does not replace training (not even close) • We are not walking through building an App • I am not debugging your source code • I will show you how to use Splunk from the command line • I will show examples of the REST API • I will show some sample work 3
- 5. Making it easy to Customize Extend
- 6. How Do Developers Use Splunk? Accelerate Dev & Test Integrate with IT Infrastructure Build Real-time Big Data Applications
- 7. Accelerate Dev & Test Every Application Developer Should Use Splunk – Get applications ready for production faster – Find and fix bugs – Trace transactions in real time – Build operational intelligence into your apps without defining a schema – Use semantic logging for better insight gathering
- 8. Unlock the power of Splunk for the entire enterprise – Search, manage and visualize Splunk data outside of Splunk Fully documented and supported REST API – Extremely flexible, over 170 endpoints Fully documented and supported SDKs for Python, Java, JavaScript, PHP, Ruby, and C# – Support for popular, open languages Integrate with IT Infrastructure
- 9. Splunk offers a fully-integrated platform – Collection, storage, query language, visualization “outof-the-box” – Real-time insights: clickstream analysis, IT earlywarning systems, security and fraud protection – Enterprise-grade scale and access control – Support for popular, open languages Build Real-time Big Data Applications
- 10. Performance Testing for Dev Ops “We use Splunk to monitor the full software development cycle - from version control commits, to continual integration builds, to agile issue tracking tools, to continual deployment stats. All of the data combined can be used to illustrate the health of development efforts in real-time.” Dan Cundiff Production Engineer • Splunk monitors activity from code commit through production • Target uses Splunk to ensure real-time health and stability of continuous integration for DevOps agility and responsiveness 10
- 11. Running Real-time Searches with the Java SDK “Splunk lets us find and fix issues on a customer’s DVR before anyone makes a phone call.” Travis Parchman Operations • • 11 Support Reps issue real-time searches from a custom CSR app to inspect customer DVR health Proactive maintenance leads to lower support costs for Comcast and higher customer satisfaction
- 12. Powering customer-facing apps with Splunk “The Splunk Python SDK provides us with a familiar developer environment to build an application on Splunk that provide metrics and comparables to our entire customer base.” Bill Matthews CTO • • MSP providing network monitoring and security management Using the Python SDK to build custom dashboards for customers powered by data from Splunk 12
- 13. What You Need to get started • • • • • Splunk Data Text Editor or IDE Documentation on dev.splunk.com SDKs on dev.splunk.com & https://github.com/splunk (* They are FREE!) 13
- 15. The Splunk Platform Inputs, Apps, Other Content Operational Intelligence Platform UI Content SDK REST API Core Functions User and Developer Interfaces Core Engine Search Processing Language Indexing Collection 15
- 16. Powerful Platform for Enterprise Developers Build Splunk Apps Web Framework Extend and Integrate Splunk Simple XML SDKs Data Models JavaScript Ruby Java JavaScript C# PHP Python Search Extensibility Django REST API 16 Modular Inputs
- 17. Hunk: Powerful Developer Platform on Hadoop Web Framework Simple XML SDKs Data Models JavaScript Ruby Java JavaScript C# PHP Python Search Extensibility Django REST API Hadoop Storage 17
- 18. Web Framework
- 19. Splunk Web Framework Brief History / Overview • Available as a stand-alone web • • • • framework in 5.0 Built into Splunk 6.0 Created for web developers Pre-built components Splunk dashboard styles 19
- 20. Splunk Web Framework Familiar Technologies Web Framework - Code with JavaScript & Django/Python - HTML5/CSS/JS Support - Built on JQuery & Backbone.js Simple XML / HTML Flexible and Extensible JavaScript - Create custom layouts - Integrate visualizations like Sankey charts, heat maps and bubble charts - SimpleXML to HTML Conversion Django 20
- 21. Get More Familiar with Web Framework Web Framework App - http://apps.splunk.com/app/1613/ Web Framework Components: • • • • • • • Search Managers Charts Tables Forms Search Manager Controls Map Data Template 21
- 22. Using the Web Framework Toolkit components Where to find the components: – $SPLUNK_HOME/etc/apps/splunk_wftoolkit/django/splunk_wftoolkit/static/spl unk_wftoolkit/components/ When the toolkit is not installed: – Include components in your app – Remember to keep them updated! When the toolkit is installed: – Reference components in the toolkit Each of the following examples also include Example Code 22
- 23. New Toolkit Components Sankey Chart Bubble Chart 23
- 24. New Toolkit Components Force-Directed Graph Calendar Heat Map 24
- 25. New Toolkit Components Parallel Coordinates Parallel Sets 25
- 26. New Toolkit Components Sunburst App also contains: • Visualization examples – Django – JavaScript • Layout templates – Django – JavaScript 26
- 27. REST API
- 28. The REST API (and SDKs) Search Visualize Manage Create and run searches from other applications Integrate search results with other applications using custom visualizations Add/Delete Users 28
- 29. The Splunk REST API Exposes an API method for every feature in the product – Whatever you can do in the UI – you can do through the API – Run searches – Manage Splunk configurations API is RESTful – – – – – Endpoints are served by splunkd Requests are GET, POST, and DELETE HTTP methods Responses are Atom XML Feeds Versioning introduced in Splunk 5.0 Search results can be output in CSV/JSON/XML/raw 29
- 30. What else about the REST API? Common HTTP Status Codes – Returned after all endpoint requests (All your favorites including 404) Authentication Methods – HTTP Header – HTTP Basic Global pagination and filtering – Returned lists of objects adhere to a standard interface Object Sharing and Permissions – All endpoints that list user objects support object sharing and Access Control List (ACL) 30
- 31. Where can you make API requests? A terminal Web browser Any code Lets see some examples!!! 31
- 32. SDKs and Tools
- 33. Overview of the Splunk SDKs Currently, Splunk has SDKs for these languages: • • • • • • What you can do • Integrate with 3rd-party tools • Log directly to Splunk • Integrate Splunk search results into Python Java JavaScript PHP Ruby C# your application • Extract data for archiving • Build a UI on the web stack of your choice • ...and so much more 33
- 34. What the Splunk SDKs do for you Handling HTTP access including certs Authenticating – a session key can be used for subsequent requests Managing namespaces Simplifying access to REST endpoints Building the correct URL for an endpoint Displaying simplified output for searches Simplification! 34
- 38. Logging Events via HTTP REST 38
- 40. Blocking, One Shot, Real Time Searches // Always block until results are ready. queryArgs.put("exec_mode", "blocking"); Job job = service.getJobs().create(query, queryArgs); // Execute the oneshot query, which returns the stream (i.e. there is // no search job created, just a one time search) InputStream stream = service.oneshotSearch(query, queryArgs); // Always set real time search mode; No need to wait on a job queryArgs.put("search_mode", "realtime"); 40
- 42. Keyword Statistics Top Hosts, Sources, Sourcetype by keyword. Baselines, Slope, Donut Chart, Outliers, and Predict 42
- 43. JMX Poll local or remote JMX Management Servers from Mbean Statistics 43
- 45. Anonymous Mood Generator • Moral and mood towards different projects and tasks based on Google Forms Survey. 45
- 46. Closer look Network device Modular Input Custom Splunk Command Not on apps.splunk.com yet, they are beta – Available on https://github.com/jamesdon 46
- 48. The Splunk Developer Community Splunkbase 48
- 49. Where to Go for More Info • Tutorials, Code Samples, Getting Started, Downloads – http://dev.splunk.com/ & http://dev.splunk.com/hunk • Support – https://www.splunk.com/page/submit_issue • GitHub – https://github.com/splunk/ • Twitter – https://twitter.com/splunkdev • Blog – http://blogs.splunk.com/dev/ 49
- 50. Thank You
Editor's Notes
- The Splunk Development Platform makes it easy to customize and extend Splunk to make the most out of your IT investments. Developers use Splunk in 3 ways
- Core Splunk “out of the box” increases the speed and efficiency of application development, testing and provides proactive monitoring and analytics for applications in productionIn your org you probably already use Splunk for App managementBut Splunk is a great tool for App Development as well“Code isn’t complete unless it’s Splunk friendly.”Splunk is a great solution for both dev and test, from the unit level to full integration
- The Splunk Developer Platform allows you to deliver greater operational agility throughout their organization by making it easy to integrate data from Splunk with other applications. Splunk provides a fully documented and supported REST API with over 170 endpoints that let developers programmatically index, search and visualize data in Splunk from any application. Splunk’s SDKs let developers integrate Splunk data with other applications across the enterprise, from custom-built mobile reporting apps to off-the-shelf CRM solutions, using familiar languages and frameworks.
- Build Real-time Data Applications with the SDKs / REST APISplunk is a fully-integrated platform that delivers rapid “time-to-value” to developers. Many of our customers are building robust applications on Splunk today that deliver real-time business insights like clickstream analysis, IT early-warning systems, security and fraud protection at a scale that their businesses demand.This is not limited to everyday customers, as vendors are using Splunk’s analytics engine to power their own Aps.
- Target Splunk’s data from Jenkins (build server) to monitor their continuous deployment/dev ops.
- Splunk Use Case: Comcast has many different Splunk use cases. Comcast’s X1 Platform Device Manager CSR app uses the Java SDK to issue (one-shot) queries (complex, with multiple sub-queries) to find errors of the last 60 minutes (needs to be realtime – submits the query syntax live). Looking for DVR attempts/failures, Baud attempts/failures, etc. Proactive investigation of customer DVRs - reps need to see if customers are actually using their DVR (I.e. Don't want to reboot a box if customers are watching TV). The goal is to fix the error before anyone makes a phone call (saving on support operations costs and increasing customer satisfaction).Before Splunk, DVR box daemon's were batch polled once a day, which was much less effective identifying errors, leading to higher call volume and lower sat. Business Value: Lower support costs& increased customer satisfaction
- IDE - integrated development environment
- As the Splunk platform evolves you can see how the Development capabilities have expanded to enable developers to better build on Splunk’s core capabilities. Let’s double click into the Developer platform….Whatdoes this platform look like?The platform consists of 2 layer:A core engine and an interface layerOn top of the platform you can’t run a broad spectrum of content that supports use casesUse cases range from application mgmt. and IT operations, to ES and PCI compliance, to web analyticsThe core engine provides the basic services for real time data input, indexing and search as well alerting, large scale distributed processing and role based accessThe Interface layer consist of the basic UI for search, reporting and visualization– it contains developer interfaces, the REST API and SDKsThe SDKs provide a convenient access to core engine services in a variety of programing language environments. These programmatic interfaces allow you to either:extend Splunkintegrate Splunk with other applicationsbuild completely new applications from scratch that require OI or analytical services that Splunk provides
- BUILD SPLUNK APPSThe Splunk Web Framework makes building a Splunk app looks and feels like building any modern web application. The Simple Dashboard Editor makes it easy to BUILD interactive dashboards and user workflows as well as add custom styling, behavior and visualizations. Simple XML is ideal for fast, lightweight app customization and building. Simple XML development requires minimal coding knowledge and is well-suited for Splunk power users in IT to get fast visualization and analytics from their machine data. Simple XML also lets the developer “escape” to HTML with one click to do more powerful customization and integration with JavaScript. Developers looking for more advanced functionality and capabilities can build Splunk apps from the ground up using popular, standards-based web technologies: JavaScript and Django. The Splunk Web Framework lets developers quickly create Splunk apps by using prebuilt components, styles, templates, and reusable samples as well as supporting the development of custom logic, interactions, components, and UI. Developers can choose to program their Splunk app using Simple XML, JavaScript or Django (or any combination thereof).EXTEND AND INTEGRATE SPLUNKThe Splunk SDKs include documentation, code samples, resources and tools to make it faster and more efficient to program against the Splunk REST API using constructs and syntax familiar to developers experienced with Java, Python, JavaScript, PHP, Ruby and C#. Developers can easily manage HTTP access, authentication and namespaces in just a few lines of code. Developers can use the Splunk SDKs to: - Run real-time searches and retrieve Splunk data from line-of-business systems like Customer Service applications - Integrate data and visualizations (charts, tables) from Splunk into BI tools and reporting dashboards- Build mobile applications with real-time KPI dashboards and alerts powered by Splunk - Log directly to Splunk from remote devices and applications via TCP, UDP and HTTP- Build customer-facing dashboards in your applications powered by user-specific data in Splunk - Manage a Splunk instance, including adding and removing users as well as creating data inputs from an application outside of Splunk- Programmatically extract data from Splunk for long-term data warehousingSplunk Enterprise offers search extensibility through: - Custom Search Commands - Scripted Lookups- Scripted Alerts- Search Macros
- Hunk: The Most Powerful Platform for Building Big Data Apps on HadoopHunk is the fully-featured platform that provides rapid exploration, analysis and visualization of your data at rest in Hadoop. It’s based on years of experience building big data products deployed at thousands of Splunk customers and drives dramatic improvements in the speed and simplicity of getting insights from big data in Hadoop. Hunk works with Apache Hadoop or the Hadoop distribution of your choice including first-generation MapReduce and YARN. Hunk also provides a rich developer platform including an integrated Web Framework and Software Development Kits (SDKs) for the world’s most popular development languages. With Hunk, you can build apps powered by data stored in Hadoop Distributed File System (HDFS) as well as integrate data from HDFS into other applications and systems without having to manually program MapReduce jobs.Hunk provides a rich developer environment that enables you to build powerful enterprise Big Data apps that deliver business insights like clickstream analysis, deep customer behavioral modeling and security analysis at enterprise-grade scale using the languages, frameworks and tools you know. Applications built on Hunk can deliver segmented, secure views of data through a highly customizable, flexible interface. The Splunk Web Framework makes building an app on top of Hadoop look and feel like building any modern web application. Developers can also integrate and extend the power of Hunk. The REST API and Software Development Kits (SDKs) for Java, JavaScript, Python, C#, Ruby and PHP, enable you to integrate data and functionality from Hunk into other applications across the enterprise, from custom-built mobile reporting apps to Web Parts in Microsoft SharePoint.Hunk offers ad hoc exploration, analysis and visualization of historical data at rest in Hadoop. Dynamically query data in HDFS or write a custom search script in a few lines of Python without having to cobble together numerous other projects and components or set up MapReduce. Hunk utilizes the Splunk Search Processing Language (SPL™), the industry-leading method to enable interactive data exploration across large, diverse data sets. With Hunk’s schema-on-the-fly, users are not limited or constrained by rigid schemas and can immediately query and interrogate raw data in Hadoop through visual interactions and SPL for deeper analysis. You can also expand the search language to customize commands to perform custom processing or calculations and trigger a shell script or batch file with scripted alerts. You can write custom search commands in Python that take data in standard input (stdin) and output data on standard output (stdout). Once that Python script has been written, all it takes is a simple configuration setting to add the custom search command to the Hunk search pipeline. Additionally, customers with both Splunk Enterprise and Hunk licenses can search across data stored both in Hadoop and in native indexes in Splunk Enterprise – all in the same search.
- You can use the new frame work in the latest two versions of Splunk. Splunk as a product is ever evolving.
- Splunk is a fully-integrated platform that delivers rapid time-to-value to application developers. Developers can build robust applications on Splunk that deliver real-time business insights like clickstream analysis, IT early-warning systems, security and fraud protection at enterprise-grade scale using the languages, frameworks and tools that they know and love. Applications built on Splunk can deliver segmented, secure data to customers in any UI, powering your core product or service with real-time operational intelligence making Splunk more valuable and more relevant to more users. The Splunk Web Framework makes building a Splunk app looks and feels like building any modern web application. The Simple Dashboard Editor makes it easy to build interactive dashboards and user workflows as well as add custom styling, behavior and visualizations. Simple XML is ideal for fast, lightweight app customization and building. Simple XML development requires minimal coding knowledge and is well-suited for Splunk power users in IT to get fast visualization and analytics from their machine data. Simple XML also lets the developer “escape” to HTML with one click to do more powerful customization and integration with JavaScript. Developers looking for more advanced functionality and capabilities can build Splunk apps from the ground up using popular, standards-based web technologies: JavaScript and Django. The Splunk Web Framework lets developers quickly create Splunk apps by using prebuilt components, styles, templates, and reusable samples as well as supporting the development of custom logic, interactions, components, and UI. Developers can choose to program their Splunk app using Simple XML, JavaScript or Django (or any combination thereof).Use Simple XML for fast, lightweight dashboard building as well as add custom styling, behavior and visualizationsUse JavaScript for client-side development Use Python and the Django framework for server-side developmentAdvanced XML?
- Splunk has published a Web Framework App to make getting more familiar with it much more simple. Maps are now part of Splunk 6.0Search managers correspond to your Splunk search jobs, saved reports, and post-process searches.Splunk views are used to visualize data, accept form input, and control searches.Tokens and data binding provides a handy mechanism for you to bind values together from searches and views to keep them in sync.Drilldown actions let you customize the drilldown behavior when users click on views.Splunk dashboard styles are used by HTML dashboards for layout, but you can use the dashboard stylesheet in your apps too.DataTemplate view with an embedded template using Django
- The Web Framework Toolkit includes a number of new, reusable components based on open-source libraries. These toolkit components are examples that demonstrate how to build your own custom components for your own apps, as well as supported add-ons for others to include in their apps.
- Overall the
- The Splunk SDKs empower developers to deliver greater operational agility throughout the enterprise by making it easy to integrate data from Splunk with other applications. Splunk provides a fully-documented and supported REST API with nearly 200 endpoints that let developers programmatically index, search and visualize data in Splunk from any application. Splunk’s SDKs, built on that API, make it easy for developers to integrate data from Splunk with other applications across the enterprise, from custom-built mobile reporting apps to off-the-shelf CRM solutions. Splunk offers SDKs for Python, Java, JavaScript, PHP, Ruby and C#. Developers can use the Splunk SDKs with Hunk to: Run searches and retrieve Splunk data from line-of-business systems like Customer Service applications Integrate data and visualizations (charts, tables) from Splunk into BI tools and reporting dashboardsBuild customer-facing dashboards in your applications powered by user-specific data in Splunk Manage a Splunk instance, including adding and removing users
- IndexLog directly to Splunk (TCP, UDP, HTTP)SearchIncluding saved searchesExtract data from SplunkVisualizeIntegrate search results with third-party reporting tools, portals and other custom applicationsManageAdd/remove users and rolesCreate inputs
- 2 main categories – Search and Manage
- Although you can use the REST API directly, you can also use the Splunk SDKs to interact with Splunk. Essentially, these SDKs are wrappers around the REST API that do a lot of the work for you, such as:
- All communication to the Splunk REST API is through the Service object.
- In Splunk, every object you create is tied to an owner and an app – you can manage access control via specifying namespaces.