Using the Splunk Java SDK

5,312 views

Published on

Learn the latest and greatest on the Splunk Java SDK. Demos, Code, Code and more Code.

Published in: Technology, Education
0 Comments
5 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
5,312
On SlideShare
0
From Embeds
0
Number of Embeds
201
Actions
Shares
0
Downloads
157
Comments
0
Likes
5
Embeds 0
No embeds

No notes for slide
  • For those searches that stream the results (oneshot and export), the search results are not saved. If the stream is interrupted for any reason, the results are not recoverable without running the search again.
  • There is code in the develop branch (which we should probably push into main before .conf) that obviates the need for job.refresh()isDone() and isReady() refresh behind your back.
  • In order to get all events, you have to use the export endpoint. But the export endpoint has different behavior than a normal job. An export cannot be "restarted" when getting events if the network hiccups. A search job can just do another getResults() with the appropriate offset — this is because the export endpoint doesn't save the results like a search job does. But a search job has a limited number of events it will store on the server — which can be affected by status_buckets — but there is no way to guarantee the upper limit. With the default status_buckets we can get to 500K events. Itay and I experimented with hundreds of stratus_buckets but were only to get up to about 1M events, out of 13M available events.
  • Using the Splunk Java SDK

    1. 1. Using the Splunk Java SDK Presented by Damien Dallimore Developer Evangelist at SplunkCopyright © 2012 Splunk Inc.
    2. 2. About me• Developer Evangelist at Splunk since July 2012 • http://dev.splunk.com • http://splunk-base.splunk.com • Slides available for my “Splunking the JVM” session• Splunk Community Member • Splunk4JMX • SplunkJavaLogging • SplunkBase Answers• Splunk Architect and Administrator• Coder, hacker, architect of Enterprise Java solutions around the globe in many different industries(aviation, core banking, card payments etc…)• Yes, I do have an accent , so please restrain all your sheep, Lord of the Rings and Kim Dotcom heckles until beer o’clock  2
    3. 3. Agenda• Overview of the Splunk Platform• REST API & SDKs• Java SDK overview• Code, Code, Code !• Thinking outside the Square • Alternate JVM Languages • Making it easier for developers to log to Splunk • Splunk is not just for Production• Questions (feel free to yell out at any time also)
    4. 4. Splunk & Developers Accelerate development &Machine Data SplunkUI Custom/Existing testing with proactive (Splunk Apps) Applications monitoring SDKs Search, chart and graph Save and schedule searches as alerts Integrate data from Splunk into Export search results your existing IT environment for Manage inputs and indexes Add & remove users and roles operational visibility REST APIs Quickly deliver real-time Splunkd business insights from Big Data outside of IT 4
    5. 5. REST API & SDKs
    6. 6. What you can do with the SDKs & API • Integrate with third-party reporting tools and portals • Log directly to Splunk • Integrate Splunk search results into your application • Extract data for archiving, compliance • Build a custom UI of your choice 6
    7. 7. Splunk REST API• Exposes an API method for every feature in the product • Whatever you can do in the UI – you can do through the API. • Run searches • Manage Splunk configurations• API is RESTful • Endpoints are served by splunkd • Requests are GET, POST, and DELETE HTTP methods • Responses are Atom XML Feeds • JSON coming in 5.0 • Versioning coming in 5.0 • Search results can be output in CSV/JSON/XML 7
    8. 8. Language SDKs• The SDKs make it easier for you to use the raw REST API , abstracting away much of the lower level plumbing, so you can instead just focus on developer productivity • Handling HTTP access • Authenticating • Managing namespaces • Simplifying access to REST endpoints • Building the correct URL for an endpoint • Displaying simplified output for searches • Input of data to a Splunk index• Python, Java, Javascript in beta - Supported• PHP available now!• Still study the core REST API though , if you’re anything like me you like to know what is going on under the hood 8
    9. 9. Java SDK overview
    10. 10. Java SDK Design Principles• Provide comprehensive coverage of the REST API• Have a 1:1 mapping of endpoint to class in the SDK• Provide implementation that felt intuitive to a Java developer• Lowest common denominator for build – ANT (Any maven people out there?)• Project support for Eclipse and IntelliJ – to ease getting started 10
    11. 11. Get the Java SDK setup• Open sourced under the Apache v2.0 license• Clone from Github : git clone https://github.com/splunk/splunk-sdk-java.git• Current release status is “beta”• Project level support for Eclipse and Intellij IDE’s • I use Eclipse with the eGit plugin• Pre-requisites • JRE 6+ • Ant (builds, javadoc generation) • Splunk installed• Run the unit tests and examples • Setup a “.splunkrc” file in your user’s home directory • Run an Ant build • Run examples with the command line wrappers • Run the Junit tests from Ant or within your IDE 11
    12. 12. Key Java SDK Concepts• Namespaces • owner : splunk username • app : app context • sharing : user | app | global | system • Defaults to current user and default app• Service class • Instantiate an object to connect and login • Entry point for REST API calls• Client/Server state • Need to maintain state explicitly • update() : to push changes to splunkd • refresh() : to get changes from splunkd 12
    13. 13. Java SDK Class Model HTTPService Resource Service ResourceCollection Entity EntityCollection Application Index Input InputCollection SavedSearchCollection• Collections use a common mechanism to create and remove entities• Entities use a common mechanism to retrieve and update property values, and access entity metadata• Service is a wrapper that facilitates access to all Splunk REST endpoints 13
    14. 14. public String codeTime(){ return “Lets Rock n Roll”;}
    15. 15. Connecting / Authenticating 15
    16. 16. Simple Entity Retrieval 16
    17. 17. Logging Events via HTTP RESTUses receivers/simple endpoint Uses receivers/stream endpoint 17
    18. 18. Logging Events via Raw TCPIf you don’t already have a TCP port listening, simply create one via the REST APISetup Log to Splunk Teardown 18
    19. 19. Searching Overview• Search query • a set of commands and functions you use to retrieve events from an index or a real-time stream , "search * | head 10".• Saved search • a search query that has been saved to be used again and can be set up to run on a regular schedule• Search job • an instance of a completed or still-running search operation.Using a search ID you can access the results of the search when they become available. Job results are saved for a period of time on the server and can be retrieved• Search Modes • Normal : asynchronous , poll job for status and results • Blocking : synchronous , a job handle is returned when search is completed • Oneshot : synchronous , no job handle is returned, results are streamed • Export : synchronous, not a search per say, doesn’t create a job, results are streamed oldest to newestHeaps more juicy examples here : http://dev.splunk.com/view/SP-CAAAEHQ 19
    20. 20. Blocking SearchesA Job is created No Job is created 20
    21. 21. Non-Blocking Search 21
    22. 22. Non-Blocking Search (with Paging) • “maxresultrows” in Splunk config default 50K • Not recommended to change this • If result set > 50K , then page through results 22
    23. 23. Realtime Search 23
    24. 24. Saved Search 24
    25. 25. Processing CSV/JSON/XML results 25
    26. 26. Client/Server State 26
    27. 27. Namespaces 27
    28. 28. Thinking outside the square
    29. 29. Alternate JVM LanguagesScala Groovy ClojureJavascript(Rhino) JRuby PHP(Quercus)Ceylon Kotlin Jython We don’t need SDK’s for these languages , we can just use the Java SDK ! 29
    30. 30. Scala “SDK” 30
    31. 31. Groovy “SDK” 31
    32. 32. SplunkJavaLogging• A logging framework to allow developers to as seamlessly as possible integrate Splunk best practice logging semantics into their code.• Custom handler/appender implementations(REST and Raw TCP) for the 3 most prevalent Java logging frameworks in play. Splunk events directly from your code. • LogBack • Log4j • java.util.logging• Implementation of the SPLUNK CIM(Common Information Model) 32
    33. 33. Developers just log as they are used to Better A-HA2012-08-07 15:54:06:644+1200 name="Failed Login" event_id="someID" app="myapp" user="jane" somefieldname="foobar" 33
    34. 34. Logging Framework takes care of the Splunk transport , REST or Raw TCP 34
    35. 35. <barf>Typical Java Stacktraces in logs</barf> 35
    36. 36. SplunkJavaLogging is your friend 36
    37. 37. Java Stacktraces in Splunk 37
    38. 38. Use Splunk in dev/test => better quality delivered to prod Testing Tools • It’s not good enough to assert that your Splunk software is production ready because the Java SDK load test “doesn’t make it fall over” Execute Tests • Splunk the app’s machine data throughout REST Perform test assertions the lifecycle of the test • Via a Splunk SDK, enrich your test harness Splunk REST% TCP /% pass/fail assertions with outputs of SplunkYour Java SDK SplunkD searchesApp Other Universal% • Catch malignant code tumors that may have Metrics Forwarder flown under the radar that your harness alone couldn’t possibly know about. • Grinder + Java SDK + Assertions 38
    39. 39. Contact DetailsAlways more than happy to be contacted for questions,feedback, collaborations, ideas that will change the world etc…Email : ddallimore@splunk.comSplunkBase: damiendGithub: damiendallimoreTwitter : @damiendallimoreBlog : http://blogs.splunk.com/devSplunk Dev Platform Team : devinfo@splunk.com 39
    40. 40. LinksGists for all code examples : https://gist.github.com/damiendallimoreJava SDK Homepage : http://dev.splunk.com/view/java-sdk/SP-CAAAECNJava SDK Github repository : https://github.com/splunk/splunk-sdk-javaSplunkJavaLogging : https://github.com/damiendallimore/SplunkJavaLoggingSplunk Best Practice Logging : http://dev.splunk.com/view/logging-best-practices/SP-CAAADP6Splunk REST API :http://docs.splunk.com/Documentation/Splunk/latest/RESTAPI/RESTcontents 40
    41. 41. //Thanks for coming !System.exit(5150);

    ×