Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Similar to Splunk Architecture overview
Similar to Splunk Architecture overview (20)
Recently uploaded
Recently uploaded (20)
Splunk Architecture overview
- 1. 1 SPLUNK OVERVIEW ALEXANDER FOK BIG DATA ARCHITECT FEBRUARY 2017
- 2. 2 • What is Splunk? • Splunk Main Functionality • Why Splunk? • Demo Agenda
- 3. 3 • Real Time logs collection, indexing and data analytics • Time Series data management • Search query language ̶ tail –f ALL_LOGS*| grep “WHATEVER YOU NEED” ̶ tail –f SOME_LOGS| grep “WHATEVER YOU NEED”| count by InterestingField ̶ Commands Pipe ̶ tail –f SOME_LOGS| grep “WHATEVER YOU NEED”| count by InterestingField Splunk Main Functionality
- 4. 4 • Presentation level – graphs, tables, etc • Historical analysis • Automation capabilities • APIs ̶ REST ̶ Command line Data Exploration and Visualization Capabilities
- 5. 5 • Rolls Royce in the field • Proven field success • Flexible, User friendly, modern tool • Enterprise grade – users access management, security, multitenant platform, data retention management policy •Reach Echo system •Splunk appstore – splunkbase.splunk.com Why Splunk?
- 6. 6 •Strong Visualization Capabilities – reports, dashboards •Infinite scale – up to hundreds of TB logs per day •Strong Post processing capabilities - Calculated and Extracted Fields •Various Optimizations ̶ Frequent Reports precalculation ̶ Lookup tables ̶ Field Tags •Advanced Data Models - CIM Why Splunk?
- 7. 7 sourcetype=mysql_config OR sourcetype=remedy_changeticket | dedup _raw, User | transaction TicketId, User | eval hasTicket = if(eventcount > 1, "Yes", "No") | rename PrevPropValue as "Original_Value", NewPropValue as "New_Value", hasTicket as "Change_Ticket" | fields _time, User, Property, "Original_Value", "New_Value", "Change_Ticket"
- 8. 8 • Web logs • Log4J, JMS, JMX • .NET events • Code and scripts • Configurations • syslog • SNMP • netflow • Configurations • Audit/query logs • Tables • Schemas • Hypervisor • Guest OS, Apps • Cloud • Configurations • syslog • File system • ps, iostat, top • Registry • Event logs • File system • sysinternals Logfiles Configs Messages Traps Alerts Metrics Scripts TicketsChanges Linux/UnixWindows NetworkingDatabasesApplications Virtualization & Cloud • Click-stream data • Shopping cart data • Online transaction data Customer Facing Data Outside the Datacenter • Manufacturing, logistics… • CDRs & IPDRs • Power consumption • RFID data • GPS data No predefined schema, no custom connectors, no RDBMS, no need to filter/forward. Splunk – The Big Picture 8
- 10. 10 Splunk’s MapReduce-based Architecture 1 0 Chunk 1 Chunk 2 Chunk 3 Chunk 4 Chunk 1 Chunk 2 Chunk 3 Chunk 4 Chunk 1 Chunk 2 Chunk 3 Chunk 4 Search Head map map map map map map map map map Answer reduce Server 1 Server 2 Server N time
- 11. 11 •Events, Indexes, Fields – key value pairs, columns •Index Time ̶ events are processed, classified, time stamp is extracted indexed ̶ Predefined Fields are extracted ̶ events can be enriched ̶ Events can trigger logic -> alerts, reports, dashboards updates etc •Search Time ̶ events are searched ̶ fields are extracted or calculated ̶ transactions are closed ̶ Visualizations can be built Splunk Typical WorkFlow
- 12. 12 •Show events counts by SFlow •SFlow|stats count by SFlow •| transaction SAUPID startswith="Product Start" endswith="Product End" Demo
- 13. 13 What is An App? • Terminology • Apps – A workspace that solves a specific use case with a navigable view • Add-on – A reusable Splunk component that does not contain a view • Example • Splunk for Cisco Security is an App • The collection of field extractions/sourcetypes/transforms/eventypes that map raw firewall logs is an Add-on
- 14. 14 •CIM – Common Information Model •Domain centric data models – OSSEC, networking, ticket management •Data normalization •Validation •Visualization •Action generation Splunk as SIEM
- 15. 15 Marathon Tel Aviv 2017 – See you tomorrow
- 16. 16 Alexander Fok, Big Data Architect THANK YOU