More Related Content Similar to Sp livenyc dow jones_10_21_2010_edit Similar to Sp livenyc dow jones_10_21_2010_edit (20) Sp livenyc dow jones_10_21_2010_edit3. About Brian Guilfoyle
7 years with Dow Jones
Previously AVP Enterprise Monitoring and Design at
AllianceBernstein
Specialties include:
Enterprise cacti deployments providing high performance IT data logging and
graphing solutions
Tideway Foundation integration for configuration item (CI) discovery, reporting
and CMDB reconcilement
Washington, DC May 20, 2010 © Splunk Inc. Confidential 2010 3
4. Large Media Organization
Leading provider of global business news and information services
Consumer and Enterprise media groups service millions of customers
7000 employees
Washington, DC May 20, 2010 © Splunk Inc. Confidential 2010 4
5. Our Environment
6000+ servers globally
13,500 + source types
1,700 network devices (primarily Cisco and Juniper)
4,000+ devices feeding into Splunk
16 Splunk indexers (for various business lines)
Indexing ~100 GB/ day
Washington, DC May 20, 2010 © Splunk Inc. Confidential 2010 5
6. Wanted: New Logging Solution
Command centers need to know what’s happening before
customers do
Outlook outages could impact internal users’ productivity
Editorial sites can’t go down—direct impact to customer service and revenue
Existing log monitoring solution was not accessible, distributed, or
fully featured
Prohibitive cost per device
Washington, DC May 20, 2010 © Splunk Inc. Confidential 2010 6
7. Splunk vs. Patrol
Ability to update all agents + searches from deployment server
With Splunk we can look at all servers from one place
Splunk can handle complex, logic-based expressions
Splunk is more cost-effective in licensing for throughput versus
per agent or per user
More secure, ability to browse directly to Splunk to see
application log files
Washington, DC May 20, 2010 © Splunk Inc. Confidential 2010 7
8. Splunk for Monitoring
Splunk’s saved searches send infrastructure issues to Operations
Splunk generates significantly fewer false positives
Much easier to manage and deploy new servers due to
centralization of saved searches
Much easier to troubleshoot across all 6000 servers
Washington, DC May 20, 2010 © Splunk Inc. Confidential 2010 8
9. Splunk/Smarts Architecture TCP 9000
(splunk) TCP 9000
ESM/GTS – Splunk Architecture SMARTS Infrastructure (splunk)
Splunk Reporting
Splunk Indexer
V4.1.4
Operator Operator Operator
Smarts SAM CMG Slot Smarts SAM EMG Slot Smarts SAM CORP Slot
TCP 389 LDAP
User Searches
(Authenticate with
Indexers via
LDAP (AD)
Smarts Trap Smarts Trap Smarts Trap Splunk QA
Splunk – Distributed Search CMG EMG CORP
Deployment
TCP 8089
(Splunk Mgmt)
SNMP UDP 162
Splunk Splunk
Indexers Indexers
V4.1.4 V4.1.4
Indexer Indexer Indexer Indexer Indexer Indexer Indexer Indexer Indexer Indexer Indexer Indexer Indexer Indexer
Netscalar LB Netscalar LB
TCP 9000
Fsplunkc.dowjones.net Fsplunk.fdotc (splunk)
TCP 9000 172.25.208.249 172.27.19.205 UDP 514
Load TCP 9000
(splunk) (splunk) (syslog)
UDP 514 TCP 9000 Balancers
(splunk) TCP 9000 UDP 514
(syslog) (syslog)
UDP 514 (splunk)
(syslog) UDP 514
Splunk (syslog) Splunk
Splunk
Splunk Forwarders Forwarders
Forwarders
Forwarders (Factiva (Remote
(Corp, CMG,
(Factiva Corp) Product) Offices)
EMG)
v3.4.9 v3.4.9 v3.4.9
v3.4.9 Splunk Splunk
Forwarders Forwarders Splunk
(MW SB) (MW SEC) Forwarders
v3.4.9 v3.4.9 (HKG)
v3.4.9
Washington, DC May 20, 2010 © Splunk Inc. Confidential 2010 9
10. Scheduler Dashboard
100 – 200 Saved
Searches running on
each indexer every 3
minutes
Need to detect
problems and send
alerts ASAP
Indexers specially
tuned for this
10
Washington, DC May 20, 2010 © Splunk Inc. Confidential 2010 10
11. Charging Business Units for Splunk
Each business unit has dedicated Splunk indexer
We separate out licenses and charge based on business unit usage
Common saved searches across most LOBs
Specialized searches based on interesting application data
Secure, role-based access makes dashboards and data available
to appropriate parties
Managing deployment saved searches through indexers
Washington, DC May 20, 2010 © Splunk Inc. Confidential 2010 11
12. Managing
Licensing
“Indexing Volume”
dashboard used to
view licensing
Custom searches
to spot “flooding”
hosts
12
Washington, DC May 20, 2010 © Splunk Inc. Confidential 2010 12
13. Getting Smarts
Consolidating Smarts
data into Splunk uber
dashboard
Easier to dig in through
Splunk than through
Smarts
13
Washington, DC May 20, 2010 © Splunk Inc. Confidential 2010 13
14. What’s Next?
Real time alerting!
More on business intelligence
More on user patterns/ interaction for improved customer
experience
Washington, DC May 20, 2010 © Splunk Inc. Confidential 2010 14
15. Questions?
15
15
Washington, DC May 20, 2010 © Splunk Inc. Confidential 2010
Editor's Notes How much dataShow F5 productsStart more with the challenges of managing an F5 environment today from log management, troubleshooting, security and complianceHow does this translate to F5 Apps in their own environment?Look to ESS as an example What’s Next?Business intelligence reportingUser pattern reporting