2. Allen Tucker
Manager, HELPnet Central Systems Team
Kelly Zimmerman
Systems Administrator, HELPnet Central Systems Team
Daniel Daily
Systems Administrator, HELPnet Central Systems Team
3. Disclaimer
During the course of this presentation, we may make forward looking statements regarding future events
or the expected performance of the company. We caution you that such statements reflect our current
expectations and estimates based on factors currently known to us and that actual events or results
could differ materially. For important factors that may cause actual results to differ from those contained
in our forward-looking statements, please review our filings with the SEC. The forward-looking
statements made in the this presentation are being made as of the time and date of its live presentation.
If reviewed after its live presentation, this presentation may not contain current or accurate information.
We do not assume any obligation to update any forward looking statements we may make.
In addition, any information about our roadmap outlines our general product direction and is subject to
change at any time without notice. It is for informational purposes only and shall not, be incorporated
into any contract or other commitment. Splunk undertakes no obligation either to develop the features
or functionality described or to include any such feature or functionality in a future release.
4. • Background and Culture
• Infrastructure
• Scalable Architecture
• Multi-Tenant onboarding
• Phases
• Expansion of service
• Q&A
6. Indiana University, est. 1820
• $3.3B enterprise
• Partnered with $6B IU Health system
• 115,000 Students
• 1.3M Credit Hours per semester
• >20,000 Degrees per year
• $1.1B in Financial Aid
• $450M in research grants
• 8,000 Acres
• 882 buildings, 36M square feet
• >600,000 living Alumni
• 10,500 Faculty and Staff
8. Safeguards
IU I.T. Policy
– IT-12 list of ‘best practices’ for system management
IU Internal Audits
– In depth departmental checks for IT operations
– Alignment with IT policies
Log management in IT-12
– Success/Failed User Logons, Success/Failed File Accesses
9. Implications
Costs associated with log review
– Its overwhelming
Different log sources,
many servers
TONs of logs.
– Costly if departments DIY
– Staff time is at a premium
Admins can make much better use of your
time being impactful to their departments
17. Performance Statistics
IOPS > 1062 via Bonnie++
~270,000 events per second dense search in
smart mode
~5400 events per second sparse search in
smart mode
21. Search Head Cluster
Use of a deployer
Knowledge object replication
Ease of additional search head
expansion
Deployer
Search Cluster
22. Indexing
Deployment Server
Houses all important Splunk
configurations
– Indexer configurations
– Configuration push to 2000+ servers
(For a team that never touches the GUI) Deployment
Forwarders
27. Simplify Onboarding
Using an Onboard Script to clone our apps
• Checks user logon
• Asks for location, OS type (user input phase)
• Adds server classes and indexes
• Copies source application template
• Replaces department variables
• Assigns roles (authorize.conf, authentication.conf)
• Assigns a default_namespace
28. Overcoming Distributed IT Administration
NO ADMIN RIGHTS
– Solution: Each department gets a shared
Box folder
Splunk Universal Forwarder
Installation scripts
How to docs
MS ConfigMan (Windows) or Ansible
(Linux) helps a great deal
The magic of deploymentclient.conf
29. Script Details
• Executes the .msi silently / stops the splunk service
Defines location, OS, Department & Host –injects
them into the deploymentclient.conf file - Assigns the
deployment server
Starts the service so it can phone home
Verifies if Splunk is installed
Also configures Deploymentclient.conf
Modifies auditd.conf for splunk access
Sets ownership and permissions
Configures splunk to start on boot
Differences
• Dropping all unnecessary event IDs
• Ingesting full auditd
30. Script to deploy to many
One script to install them all
Mass deployment of forwarder to 600+ servers remotely.
38. Down the Road
Further development IT-12 application for
departments
– Devices not checking in
– Alerting for failed user attempts
– Filter for network access vs file access
– More Linux distros
Expand UITS I.T. Ops.
– Kuali
– VMware
– Active Directory
– IIS / Apache for IU Central Web Services
– Multiple Security Apps
Large organization:
$3.3B enterprise
115,000 students
10,500 faculty and staff.
8 campuses
882 separate buildings over 8000 acres
Given the scale, what does the I.T. landscape look like?
Central enterprise I.T. organization called UITS that handles management of core services
Active Directory, Exchange, Networks, Wireless, Web Hosting, etc.
Departmental I.T. providing hands on support directly to individual departments.
109 Separate IT groups
Total servers = 5213 servers (UITS and Departmental)
Obviously this means that IU has a large cyber security attack surface. Because of this, IU has implemented some safeguards to combat these issues.
IT12 – Best Practices
Admin rights, network security, firewalling, as well as log management
There are also Internal Audits that take place.
Very public results (deans, directors, CIO, President, Board of Trustees)
unfortunately, a consistent finding is log management.
Manual log review is very costly
Time consuming
Costly to build a utility to automate it
Staff time is at a premium where they could be more helpful by direct support and innovation in their direct department.
Departments - “If we have to review our logs because of IT policy, UITS should provide enterprise utility to make it easier”
- We set out to provide a clear cut time saving utility that directly aligns with IT policy in order to wipe the slate clean on log review being an issue.
3 years ago, HELPnet brought hardware log management online for internal use to meet IT12 requirements
Started offering it at a small scale to other departments.
Proposal approved in August by CIO Brad Wheeler to hopefully reduce these findings from Internal Audit.
Indiana University has a world class Storage and Virtualization team
- vmware innovation award.
- pushing the boundries of what virtualization can do
Indiana University has a world class Storage and Virtualization team that has earned vmware innovation awards.
Multi-tier storage
(low tier)
This is our initial infrastructure based on the initial service offering and based off of our current deployment needs
Discuss search heads / Indexers
Discuss locations
Explain what IOPS are and what Bonnie ++ is
With dozens and dozens of configuration files and the ability to expand with virtualized hardware, what does Splunk offer to help manage these servers?
So I am going to talk about the things we did to make our admin of splunk easier
How we:
Save Countless hours of staff time (primary example being an onboarding script)
Reduce the administration of end points (no need to touch every server or forwarder in the environment)
Quickly recover servers and stand up new servers --be it an additional SH or IDXR
Take advantage of this extra time?
--More hands on time with departments and development of apps ( spend time getting data into Splunk) esp non it12 data
--Training our users to be power users
So what features did we decided to introduce? first up our search head cluster- next slide
The star of your cluster is the deployer
Single point of administration
It’s the best place to back up all your configurations
To deploys all your apps & configurations to the cluster
It’s the job scheduler ….Brains!
KO replication Joe power user…. (Senario)
(Gotcha: Local configurations)
Magic of this…?
File precedence … GUI configurations... plan the role of your deployer carefully
Scripting a new search head member gets a lot easier when all you have to do is spin up a new VM and assign it to the cluster (and the LB) and the let the deployer push base line configurations. Stuff replicates it’s a thing of beauty.
Don’t confuse this cluster technology with MS clusters… Next up Deployment server….next slide
Deployment server is optimal BUT its a must
It is a centralized administration point like that of the cluster deployer, however it pushes configs to indexers, and universal forwarders.
Without it you would have to administer the configurations of your splunk components one system at a time. No one has time for that!
-Adding indexers-
The real beauty is the management of forwarders. Say you have a forwarder already on a system….you can increase/decrease the amount of data coming to the indexers by just editing the settings on the deployment server, once the forwarder checks in it picks up the changes.
So what happens when an IU department comes to us to start using Splunk?
Most are seeking basic alignment services for IT12 policy alignment
Some are seeking additional splunk apps and data collection…next slide
I want to revisit this architecture slide to discuss departments at IU. We have over 50 and any of them could be potentially sending different types of data not just server security logs for it12) and they don’t want to see each others data. To facilitate this we siloed data into indexes. WHY? Next slide
SECURITY SECURITY SECURITY
Role Based Administration- AD groups. User A from dept A logs in….(scenario)
Since we are pushing this configuration from the deployment server
Indexer replication (and the lack there of)
Reporting tool not a repository tool
So how do we make deployment easier?
For IT12 -- It starts on the back end getting our app copied and configs in place for the forwarder to phone home for the first time and sync
So Dainel built a script to do the heavy lifting for us.
Confirms your logged on as Splunk
Asks for location (user input)
Asks for OS type (user input)
Adds the needed server classes (assign deployment apps in serverclass.conf)
Adds the needed indexes (assign indexes in indexes.conf- homepath, coldpath, thawedpath etc.)
Copies our application template
Replaces department variables
Assigns roles (authorize.conf)
Assigns an AD group reference (authentication.conf)
Assigns a default_namespace
Gets fancy with the cron job wiz (looks to Daniel)
Each night we push the splunk shculster-bundle cmd to prevent restarts during the day
Okay so how do we get the forwarder to the departments??
Then we have to get the forwarder to the servers….and one does not simply walk into Mordor and just receive admin rights.
Linux packaged tar.gz
PowerShell Launches Splunk Forwarder msi
No tool to deploy … no problem
Script the script deployment!
Reports saved every 2 hours
Provide a high level overview for all four policy alignment pieces
Limited access
Typical drilldown functionality takes you to a search bar
Note multi-select
File Path box
Training – Get some!
Optimizing Searches – Refine everything at the beginning, avoid heavy utilization items like transactions appends, and additional searches to searched data
Data Inputs – Make sure to use search time data manipulation vs index/forwarder time data manipulation
Reporting – Initially we tried to set up saved searches to show data, and update it every 2 hours. Use report acceleration without a schedule.
Removed the activity dropdown from the splunk bar
Added a new drop down menu with all for drilldown dashboards
Added tables
Removed all scheduled searches, and set the searches to on demand with report acceleration
Emailed reports
Added in additional exclude multi select boxes
Added behind the scenes options like optimized searches, auto populated drill down functionality, on demand searches
Main two areas for items down the road.
Usability expansions of IT12 App
Devices not checking in, Alerting, Filtering of network / console traffic
Expand IT OPS
Kuali, VMware, AD, Web Services, Security App,
First development outside of IT12 is for CAS and Shibboleth.