• Like
  • Save
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads

Views

Total Views
1,463
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
0
Comments
0
Likes
1

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide
  • We’re extending Splunk so it’s easier for your to leverage it’s capabilities using technologies you’re familiar with. We’re delivering SDKs on top of our REST API to help you integrate Splunk data with other applications. Splunk is a fully-integrated platform that delivers rapid “time-to-value” to developers. Many of our customers are building robust applications on Splunk today that deliver real-time business insights like clickstream analysis, IT early-warning systems, security and fraud protection at a scale that their businesses demand.
  • Whatdoes this platform look like?The platform consists of 2 layer:A core engine and an interface layerOn top of the platform you can’t run a broad spectrum of content that supports use casesUse cases range from application mgmt. and IT operations, to ES and PCI compliance, to web analyticsThe core engine provides the basic services for real time data input, indexing and search as well alerting, large scale distributed processing and role based accessThe Interface layer consist of the basic UI for search, reporting and visualization– it contains developer interfaces, the REST API and SDKsThe SDKs provide a convenient access to core engine services in a variety of programing language environments. These programmatic interfaces allow you to eithe:r:extend Splunkintegrate Splunk with other applicationsbuild completely new applications from scratch that require OI or analytical services that Splunk provides
  • There is code in the develop branch (which we should probably push into main before .conf) that obviates the need for job.refresh()isDone() and isReady() refresh behind your back.
  • In order to get all events, you have to use the export endpoint. But the export endpoint has different behavior than a normal job. An export cannot be "restarted" when getting events if the network hiccups. A search job can just do another getResults() with the appropriate offset — this is because the export endpoint doesn't save the results like a search job does. But a search job has a limited number of events it will store on the server — which can be affected by status_buckets — but there is no way to guarantee the upper limit. With the default status_buckets we can get to 500K events. Itay and I experimented with hundreds of stratus_buckets but were only to get up to about 1M events, out of 13M available events.

Transcript

  • 1. Copyright © 2012 Splunk, Inc.Splunk for DevelopersPaul SanfordDirector, Developer PlatformJon RooneyDirector, Developer Marketing
  • 2. AgendaAssumptionsUsing Splunk for Application DevelopmentThe Splunk Developer PlatformSplunk By Example – the Java SDKSupport and Community 2
  • 3. Assumptions
  • 4. You Are in This Session Because….• You are an experienced Splunk user (search, dashboards, sourcetyping, extracting fields)• You are a developer and want to use your development skills to customize and extend your Splunk experience• You love REST and love developing with APIs• You are interested in using SDKs to index, search, manage and visualize data in Splunk• You have http://dev.splunk.com bookmarked 4
  • 5. Using Splunk forApplicationDevelopment
  • 6. Using Splunk for Application DevelopmentAccelerate Dev & Test– Every developer should use Splunk to find and fix bugs, trace transactions in real time & build intelligence into your apps without defining a schema with semantic loggingIntegrate date from Splunk into other applications– Search, manage and visualize data in other applications with the REST API and SDKs for Java, Python, JavaScript and PHPBuild Real-time Big Data Applications– Collection, storage, query language, visualization “out-of-the-box”– Real-time insights: clickstream analysis, IT early-warning systems, security and fraud protection 6
  • 7. Splunk is Hackable!1st Splunk Hackathon: .conf2012• 50 hackers• 10 teams• 5 hours• 3 winners 7
  • 8. What You Need• Splunk• Data• Text Editor or IDE• Documentation (dev.splunk.com)• SDKs on GitHub 8
  • 9. The Splunk DeveloperPlatform
  • 10. The Splunk Platform Inputs, Apps, OtherOperational Intelligence Platform Content UI SDK Content REST APIUser and Developer Interfaces Core Functions Core Engine Search Processing Language Indexing Collection 10
  • 11. What can you do with the APIs and SDKs?Index– Log directly to Splunk (TCP, UDP, HTTP)Search– Including saved searches– Extract data from SplunkVisualize– Integrate search results with third-party reporting tools, portals and other custom applicationsManage– Add/remove users and roles– Create inputs 11
  • 12. The Splunk REST APIExposes an API method for every feature in the product– Whatever you can do in the UI – you can do through the API– Run searches– Manage Splunk configurationsAPI is RESTful– Endpoints are served by splunkd– Requests are GET, POST, and DELETE HTTP methods– Responses are Atom XML Feeds– JSON coming in 5.0– Versioning coming in 5.0– Search results can be output in CSV/JSON/XML 12
  • 13. Spring Integration Splunk Inbound Adaptor• Blocking, Non Blocking, Saved & Realtime Searches• Exporting 13
  • 14. Spring Integration Splunk Outbound Adaptor • HTTP REST Input • TCP Input 14
  • 15. Demos
  • 16. Lap around SplunkSDKs
  • 17. SDK Design Concepts• Stay true to the semantics of the particular language • E.g. Keep Python “pythonic”• Provide implementation that feels to the developer • E.g. Project, build, IDE (where applicable) support• Cover REST API endpoints based on use cases of language • E.g. Java SDK has most comprehensive coverage. JavaScript has fewer management facilities• Initially stay true to REST API semantics and abstract based on feedback• Namespaces • owner: splunk username (defaults to current user) • app: app context (defaults to default app) • sharing: user | app | global | system• Service Class • Instantiate an object to connect and login • Entry point for REST API calls 17
  • 18. Java SDK• Client/Server state • Need to maintain state explicitly • update() : to push changes to splunkd • refresh() : to get changes from splunkd• Getting Started - http://dev.splunk.com/view/java-sdk/SP-CAAAECN • Open sourced under the Apache v2.0 License • Current release status is “beta” • Clone from Github : git clone https://github.com/splunk/splunk-sdk-java.git • Project level support for Eclipse and IntelliJ (git plugins available) • Pre-requisites • Splunk installed • JRE 6+ • Ant (test, build, generate javadocs) • Run the unit tests and examples • Set up a “.splunkrc” file in your user’s home directory • Ant (build, test, generate javadocs) 18
  • 19. JavaScript SDK• 2 main components • Data SDK – Manage Splunk objects, input and search data etc. • UI SDK – Includes Splunk UI components like Charting and Timeline controls• Use of native JavaScript objects • Resource, Entity and Collection objects provide the necessary abstraction• Client/Server state • Need to maintain state explicitly • update() : to push changes to splunkd • fetch() : to get changes from splunkd• Getting Started - http://dev.splunk.com/view/javascript-sdk/SP-CAAAECM • Open sourced under the Apache v2.0 License • Current release status is “beta” • Clone from Github : git clone https://github.com/splunk/splunk-sdk-javascript.git • Pre-requisites • Splunk installed • Node.js for server side scripting, building and running tests and examples • Run the unit tests and examples using node. 19
  • 20. Python SDK• 4 main modules • binding: Provides thin abstraction over raw HTTP. • client: Provides an abstraction layer over REST APIs. • results: Provides a Splunk specific streaming XML reader. • data: Converts Splunk’s Atom feed response into Pythonic structure – directory or list• Client/Server state • Need to maintain state explicitly • update() : to push changes to splunkd • refresh() : to get changes from splunkd• Getting Started - http://dev.splunk.com/view/python-sdk/SP-CAAAEBB • Open sourced under the Apache v2.0 License • Current release status is “beta” • Clone from Github : git clone https://github.com/splunk/splunk-sdk-python.git • Pre-requisites • Splunk installed • Python 2.6+ • easy_install or pip • Run the unit tests and examples • Set up a “.splunkrc” file in your user’s home directory 20
  • 21. PHP SDK• Client/Server state • Need to maintain state explicitly • update() : to push changes to splunkd • fetch() : to get changes from splunkd• Getting Started - http://dev.splunk.com/view/php-sdk/SP-CAAAEJM • Open sourced under the Apache v2.0 License • Current release status is “preview” • Clone from Github : git clone https://github.com/splunk/splunk-sdk-php.git • Pre-requisites • Splunk installed • PHP 5.2.11+ • Web Server that supports PHP (e.g. MAMP) – for running examples • PHPUnit 3.6+ - for running the unit tests • Run the unit tests and examples • Set up a “settings.default.php” file in the examples and tests directory 21
  • 22. Code Samples: JavaSDK
  • 23. Connecting / Authenticating 23
  • 24. Namespaces 24
  • 25. Logging Events via HTTP RESTUses receivers/simple endpoint Uses receivers/stream endpoint 25
  • 26. Logging Events via Raw TCPIf you don’t already have a TCP port listening, simply create one via the REST APISetup Log to Splunk Teardown 26
  • 27. Synchronous Search 27
  • 28. Asynchronous Search 28
  • 29. Paginating Results • “maxresultrows” in Splunk config default 50K • Not recommended to change this • If result set > 50K , then page through results 29
  • 30. Real-time Search 30
  • 31. Saved Search 31
  • 32. Processing CSV/JSON/XML results Results put into Hashmap 32
  • 33. Support andCommunity
  • 34. The Splunk Developer Community Splunkbase• Over 1,000 unique visitors/week to dev.spunk.com• Over 650 followers of @splunkdev 34
  • 35. Where to Go for More InfoPortal– http://dev.splunk.com/GitHub– https://github.com/splunk/Twitter– https://twitter.com/splunkdevBlog– http://blogs.splunk.com/dev/Support 35
  • 36. Thank you