Splunk Developer PlatformDamien DallimoreDeveloper Evangelist
Copyright©2013,SplunkInc.Splunk & Developers2REST APICustom/Existing ApplicationsSDKsSearch, chart and graphSave and sched...
Copyright©2013,SplunkInc.The Splunk REST API3• Exposes an API method for every feature in the product– Whatever you can do...
Copyright©2013,SplunkInc.Developer Platform4• We want to make it as easy as possible for developers to build Big Data apps...
Copyright©2013,SplunkInc.Top 3 Developer Takeaways• Every developer can use Splunk toaccelerate dev & test and gain applic...
Copyright©2013,SplunkInc.Takeaway 1: Use Splunk to accelerate dev & test6• Splunk frees you from upfront database design f...
Copyright©2013,SplunkInc.Takeaway 2: Customize and extend Splunk7Integrate data fromSplunk into existing appsand systemsBu...
Copyright©2013,SplunkInc.Takeaway 3: Splunk lets developers build big data appswith the skills they already have8• Develop...
Copyright©2013,SplunkInc.Why choose to develop on Splunk ?9• Splunk is not agnostic of its underlying data source , MapR a...
Custom Visualizations
Copyright©2013,SplunkInc.Visualizing Splunk with the SDKs11• Splunkweb has rich, but sometimes limited, visualization opti...
Copyright©2013,SplunkInc.Development Approaches12• Custom Advanced XML Modules• Incorporate into Views in SplunkWeb Apps• ...
Copyright©2013,SplunkInc.
Copyright©2013,SplunkInc.
Copyright©2013,SplunkInc.
Copyright©2013,SplunkInc.
Copyright©2013,SplunkInc.My Guiding Viz Principle17The visualization must be simple and intuitive to understand and derive...
SDK Code ExamplesSplunk SDK for Java
Copyright©2013,SplunkInc.Get the Java SDK19• Open sourced under the Apache v2.0 license• Clone from Github : git clone htt...
Copyright©2013,SplunkInc.Java SDK Class Model20ServiceResourceResourceCollection EntityEntityCollection Application IndexH...
Copyright©2013,SplunkInc.Key Java SDK use cases21• Connect and Authenticate• Manage• Input Events• Search
Copyright©2013,SplunkInc.Connect and Authenticate22public static Service connectAndLoginToSplunkExample() {Map<String, Obj...
Copyright©2013,SplunkInc.Manage23public static void getServerInfoExample() {Service splunkService = connectAndLoginToSplun...
Copyright©2013,SplunkInc.Input Events24public static void logEventToSplunkExample() {Service splunkService = connectAndLog...
Copyright©2013,SplunkInc.Semantic LoggingLog anything that can add value when aggregated, charted or further analyzedExamp...
Copyright©2013,SplunkInc.Search26• Search query• a set of commands and functions you use to retrieve events from an index ...
Copyright©2013,SplunkInc.Blocking Searches (Oneshot)27public static void simpleSearchExample() {Service splunkService = co...
Copyright©2013,SplunkInc.Blocking Searches (Export)28public static void exportSearchExample() {Service splunkService = con...
Copyright©2013,SplunkInc.Non Blocking Search29public static void searchJobExample() {Service splunkService = connectAndLog...
Copyright©2013,SplunkInc.Realtime Search30public static void realTimeSearchExample() {Service splunkService = connectAndLo...
Copyright©2013,SplunkInc.Alternate JVM Languages31Scala Groovy ClojureJavascript(Rhino) JRuby PHP(Quercus)Ceylon Kotlin Jy...
Copyright©2013,SplunkInc.Groovy32class SplunkJavaSDKWrapper {static main(args) {//connect and logindef connectionParameter...
Copyright©2013,SplunkInc.Scala33importcom.splunk.Service._importscala.collection.mutable.HashMapimportscala.collection.Jav...
Copyright©2013,SplunkInc.Contact me34Email : ddallimore@splunk.comTwitter : @damiendallimoreSkype : damien.dallimoreGithub...
Upcoming SlideShare
Loading in …5
×

Splunk Developer Platform

2,681 views

Published on

Presentation section from Splunk Live content

Published in: Technology, Education
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,681
On SlideShare
0
From Embeds
0
Number of Embeds
9
Actions
Shares
0
Downloads
67
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide
  • UsingSplunk on development and testing to improve application quality and time-to-release
  • Developers can use the Splunk SDKs to: Run real-time searches and retrieve Splunk data from line-of-business systems like Customer Service applications Integrate data and visualizations (charts, tables) from Splunk into BI tools and reporting dashboardsBuild mobile applications with real-time KPI dashboards and alerts powered by Splunk Log directly to Splunk from remote devices and applications via TCP, UDP and HTTPBuild customer-facing dashboards in your applications powered by user-specific data in Splunk Manage a Splunk instance, including adding and removing users as well as creating data inputs from an application outside of SplunkProgrammatically extract data from Splunk for long-term data warehousing
  • Ohio-based Security MSP Hurricane Labs delivers real-time security intelligence to customers using the Splunk SDK for Python. Hurricane Labs deliver’s relevant security-related data from Splunk to their customers via custom dashboards embedded in their website.San Francisco-based startup Socialize allows mobile developers to instantly add social features to their apps. More than just ratings and comments, the Socialize platform encourages mobile app users to take “social actions” within the app to drive re-engagement, retention and distribution. These “social actions” are measured, analyzed and leveraged for subsequent re-actions that increase activity and engagement in mobile apps.Socialize leverages Splunk for MapReduce and Big Data analysis. Mobile apps using Socialize create large amounts of data, averaging over 7 million API requests per day and one million actions per month. Building on the Splunk REST API to integrate at the application level for business intelligence, reporting and alerting, Socialize exposes Splunk data to its customers through highly customized dashboards.
  • Example of a silicon valley startup , tshark , mac , 3 weeks coding custom programs , I did this in 30 mins
  • Splunk Developer Platform

    1. 1. Splunk Developer PlatformDamien DallimoreDeveloper Evangelist
    2. 2. Copyright©2013,SplunkInc.Splunk & Developers2REST APICustom/Existing ApplicationsSDKsSearch, chart and graphSave and schedule searches as alertsExport search resultsManage inputs and indexesAdd & remove users and rolesSplunkUI(Splunk Apps)Machine DataEngine
    3. 3. Copyright©2013,SplunkInc.The Splunk REST API3• Exposes an API method for every feature in the product– Whatever you can do in the UI – you can do through the API– Run searches– Manage Splunk configurations• API is RESTful– Endpoints are served by Splunkd– Requestsare GET, POST, and DELETE HTTP methods– Responses are Atom XML Feeds or JSON– Versioning Support– Search results can be output in CSV/JSON/XML/Raw– Authentication is token based
    4. 4. Copyright©2013,SplunkInc.Developer Platform4• We want to make it as easy as possible for developers to build Big Data apps andcustom integrations on top of the Splunk platform• Several different language offerings, Software Development Kits (SDKs)• Javascript, Java, Python, PHP, C#, Ruby• Our SDKs make it easier to use the REST API• All Splunk functionality is accessible via our SDKs• Get Data into Splunk• Execute Splunk Searches, get data out of Splunk• Manage Splunk• Customized User Interfaces
    5. 5. Copyright©2013,SplunkInc.Top 3 Developer Takeaways• Every developer can use Splunk toaccelerate dev & test and gain applicationintelligence• The developer platform lets customerscustomize and extend the power of Splunk• Splunk lets developers build big data appswith the skills they already have
    6. 6. Copyright©2013,SplunkInc.Takeaway 1: Use Splunk to accelerate dev & test6• Splunk frees you from upfront database design for analytics• late binding schema• Developers and QA/test engineers don’t have to ask IT/Ops to get logs off machines• Role base access to all data within one console without having to log intoproduction systems• All events are indexed and accessible in real-time in one place.• Ad-Hoc real-time monitoring and historical investigation searchable from oneplace• Correlations and insights across multiple tiers.• Splunk lets you find issues quickly, so you can fix issues quickly• Integrate Splunk search results into testing assertions
    7. 7. Copyright©2013,SplunkInc.Takeaway 2: Customize and extend Splunk7Integrate data fromSplunk into existing appsand systemsBuild custom line-of-business apps poweredby SplunkDeliver Operational Intelligence to marketing, sales, customer service and otherdivisions beyond IT in the systems and apps that make sense to them.REST API & SDKs
    8. 8. Copyright©2013,SplunkInc.Takeaway 3: Splunk lets developers build big data appswith the skills they already have8• Developers can use the languages andframeworks they know and love – likePython, JavaScript, Java and PHP.• No need to write MapReduce jobs, learn Ror be some kind of scientist to build appsthat use Big Data – be a developer!Using the Python SDK to deliver customers real-time security intelligence into custom dashboardsSplunks 7 million API calls per day and exposesSplunk data to customers in their customer-facingweb app via REST API
    9. 9. Copyright©2013,SplunkInc.Why choose to develop on Splunk ?9• Splunk is not agnostic of its underlying data source , MapR algorithm optimized to Splunk index files• Real time vs Batch Jobs• Optimal for time series based data• End to End Integrated Big Data Solution• Fine grained protection of access and data using role based permissions• Data retention and aging controls• Users can submit “Map Reduce” jobs without needing to know how to code a MapR job• Get the best of many worlds ie: Splunk Hadoop Connect• Splunk integrates easily with other systems, developers can then just focus on developing against 1 single platform
    10. 10. Custom Visualizations
    11. 11. Copyright©2013,SplunkInc.Visualizing Splunk with the SDKs11• Splunkweb has rich, but sometimes limited, visualization options• You can use the SDKs to extract data from Splunk using a search, andvisualize it in an entirely custom manner• Using the Javascript SDK you can integrate with third party chartinglibrarys like Google Charts, Rickshaw, D3,three.js etc..
    12. 12. Copyright©2013,SplunkInc.Development Approaches12• Custom Advanced XML Modules• Incorporate into Views in SplunkWeb Apps• Share on Splunkbase or reuse internally• Use our new “Application Framework” (in preview mode currently)• Use our Python and Javascript SDK’s• Leverage your skills with other JS librarys (Backbone, JQuery)• Leverage the power of Django• Shareable UI components• Simple XML parser• Code your own standalone application• Use any of our SDKs to build your own solution and UI (web based, fat, mobile)
    13. 13. Copyright©2013,SplunkInc.
    14. 14. Copyright©2013,SplunkInc.
    15. 15. Copyright©2013,SplunkInc.
    16. 16. Copyright©2013,SplunkInc.
    17. 17. Copyright©2013,SplunkInc.My Guiding Viz Principle17The visualization must be simple and intuitive to understand and derive meaning from at a glance.Cool viz , but what are you telling me ?
    18. 18. SDK Code ExamplesSplunk SDK for Java
    19. 19. Copyright©2013,SplunkInc.Get the Java SDK19• Open sourced under the Apache v2.0 license• Clone from Github : git clone https://github.com/splunk/splunk-sdk-java.git• Project levelsupport for Eclipse and IntellijIDE’s• Pre-requisites• JRE6+• Ant , Maven coming• Splunk installed• Loadsof code examples• Project examplesfolder• Unit Tests• http://dev.splunk.com• http://gist.github.com/damiendallimore• Comprehensivecoverageof the REST API• Tutorialvideos availableat http://dev.splunk.com
    20. 20. Copyright©2013,SplunkInc.Java SDK Class Model20ServiceResourceResourceCollection EntityEntityCollection Application IndexHTTPServiceInputInputCollection SavedSearchCollection• Collections use a common mechanism to create and remove entities• Entities use a common mechanism to retrieve and update property values, and access entity metadata• Service is a wrapper that facilitates access to all Splunk REST endpoints
    21. 21. Copyright©2013,SplunkInc.Key Java SDK use cases21• Connect and Authenticate• Manage• Input Events• Search
    22. 22. Copyright©2013,SplunkInc.Connect and Authenticate22public static Service connectAndLoginToSplunkExample() {Map<String, Object> connectionArgs = new HashMap<String, Object>();connectionArgs.put("host", ”somehost");connectionArgs.put("username", ”spring");connectionArgs.put("password", ”integration");connectionArgs.put("port", 8089);connectionArgs.put("scheme", "https");// will login and save the session key which gets put in the HTTP Authorization headerService splunkService = Service.connect(connectionArgs);return splunkService;}
    23. 23. Copyright©2013,SplunkInc.Manage23public static void getServerInfoExample() {Service splunkService = connectAndLoginToSplunkExample();ServiceInfo info = splunkService.getInfo();System.out.println("Info:");for (String key : info.keySet())System.out.println(" " + key + ": " + info.get(key));Entity settings = splunkService.getSettings();System.out.println("nSettings:");for (String key : settings.keySet())System.out.println(" " + key + ": " + settings.get(key));}
    24. 24. Copyright©2013,SplunkInc.Input Events24public static void logEventToSplunkExample() {Service splunkService = connectAndLoginToSplunkExample();// Get a Receiver objectReceiver receiver = splunkService.getReceiver();// Set the sourcetypeArgs logArgs = new Args();logArgs.put("source", ”http-rest");logArgs.put("sourcetype", ”spring-example");// Log an event into the spring indexreceiver.log(”spring", logArgs, ”SpringOne 2GX rocks");}• Other Input transports• HTTP REST Streaming• Raw TCP Oneshot & Streaming• Raw UDP & Syslog
    25. 25. Copyright©2013,SplunkInc.Semantic LoggingLog anything that can add value when aggregated, charted or further analyzedExample Bogus Pseudo-Code:void submitPurchase(purchaseId){log.info("action=submitPurchaseStart, purchaseId=%d", purchaseId)//these calls throw an exception on errorsubmitToCreditCard(...)generateInvoice(...)generateFullfillmentOrder(...)log.info("action=submitPurchaseCompleted, purchaseId=%d", purchaseId)}• Create Human Readable Events• Clearly Timestamp Events• Use Key-Value Pairs (JSON Logging)• Separate Multi-Value Events• Log Unique Identifiers
    26. 26. Copyright©2013,SplunkInc.Search26• Search query• a set of commands and functions you use to retrieve events from an index or a real-time stream , "searchindex=spring error OR exception | head 10”• Saved search• a search query that has been saved to be used again and can be set up to run on a regular schedule• Search job• an instance of a completed or still-running search operation.Using a search ID you can access the results of thesearch when they become available. Job results are saved for a period of time on the server and can be retrieved• Search Modes• Normal : asynchronous , poll job for status and results• Realtime : same as normal, but stream is kept open a results streamed in realtime• Blocking : synchronous , a job handle is returned when search is completed• Oneshot : synchronous , no job handle is returned, results are streamed• Export : synchronous, not a search per say, doesn’t return a job handle, results are streamed oldest to newest
    27. 27. Copyright©2013,SplunkInc.Blocking Searches (Oneshot)27public static void simpleSearchExample() {Service splunkService = connectAndLoginToSplunkExample();String searchQuery = "search error OR exception| head 10";Args queryArgs = new Args();queryArgs.put("earliest_time", "-3d@d");queryArgs.put("latest_time", "-1d@d");// perform the search , blocks hereInputStream stream = splunkService.search(searchQuery, queryArgs);processInputStream(stream);}
    28. 28. Copyright©2013,SplunkInc.Blocking Searches (Export)28public static void exportSearchExample() {Service splunkService = connectAndLoginToSplunkExample();String searchQuery = "search error OR exception | head 10";Args queryArgs = new Args();queryArgs.put("earliest_time", "-1d@d");queryArgs.put("latest_time", "now");// perform the export , blocks hereInputStream stream = splunkService.export(searchQuery, queryArgs);processInputStream(stream);}
    29. 29. Copyright©2013,SplunkInc.Non Blocking Search29public static void searchJobExample() {Service splunkService = connectAndLoginToSplunkExample();String outputMode = "csv";// xml,json,csv// submit the jobJob job = splunkService.getJobs().create("search index=spring error OR fatal | head 10");while (!job.isDone()) {try {Thread.sleep(500);}catch (Exception e) {}}Args outputArgs = new Args();outputArgs.put("output_mode", outputMode);InputStream stream = job.getResults(outputArgs);processInputStream(stream, outputMode); // uses xml stream, opencsv and gson}
    30. 30. Copyright©2013,SplunkInc.Realtime Search30public static void realTimeSearchExample() {Service splunkService = connectAndLoginToSplunkExample();Args queryArgs = new Args();queryArgs.put("earliest_time", "rt-5m");queryArgs.put("latest_time", "rt");// submit the jobJob job = splunkService.getJobs().create("search index=spring exception OR error”, queryArgs);…}
    31. 31. Copyright©2013,SplunkInc.Alternate JVM Languages31Scala Groovy ClojureJavascript(Rhino) JRuby PHP(Quercus)Ceylon Kotlin JythonWe don’t need SDK’s for these languages , we can just use the Java SDK !
    32. 32. Copyright©2013,SplunkInc.Groovy32class SplunkJavaSDKWrapper {static main(args) {//connect and logindef connectionParameters = [host:”somehost",username:"spring",password:"integration"]Service service = Service.connect(connectionParameters)//get Splunk Server infoServiceInfo info = service.getInfo()def splunkInfo = [:]for (key in info.keySet())splunkInfo.put(key,info.get(key))printSplunkInfo(splunkInfo)}static printSplunkInfo(splunkInfo) {println "Info”splunkInfo.each { key, value ->println key + " : " + value}}}
    33. 33. Copyright©2013,SplunkInc.Scala33importcom.splunk.Service._importscala.collection.mutable.HashMapimportscala.collection.JavaConversions._objectSplunkJavaSDKWrapper{def main(args:Array[String]) = {//connectand loginvalconnectionArgs= HashMap[String,Object]("host"->”somehost”,"username"->”me”,"password"->”foo")valservice= connect(connectionArgs)//get SplunkServerinfovalinfo= service.getInfo// Scala/JavaconversionvaljavaSet= info.keySetvalscalaSet= javaSet.toSet//print out SplunkServerinfofor (key <- scalaSet)println(key+ ":"+ info.get(key))}}
    34. 34. Copyright©2013,SplunkInc.Contact me34Email : ddallimore@splunk.comTwitter : @damiendallimoreSkype : damien.dallimoreGithub : damiendallimoreSplunkbase : damiendSlideshare : http://www.slideshare.net/damiendallimoreBlogs : http://blogs.splunk.com/devWeb : http://dev.splunk.com

    ×