Splunk provides a fully integrated platform for collecting, analyzing, and visualizing machine data from any source. The Splunk App for AWS integrates with AWS services like CloudTrail, Config, and VPC Flow Logs to provide visibility into AWS environments. It collects data from AWS SQS queues and S3 buckets using the AWS SDK. Setting up the Splunk App for AWS requires installing the Splunk Add-on for AWS and configuring permissions for AWS services. Users add their AWS account, and then must wait 5-10 minutes for dashboards and reports to populate with AWS log data.

1. Copyright © 2015 Splunk Inc.
Splunk For the Cloud,
in the Cloud
Richard Smith
Manager, Global Strategic Alliances
Australia and New Zealand

2. 2
Make machine data accessible,
usable and valuable to everyone.
2

3. Big Data Comes from Machines
Volume | Velocity | Variety | Variability
GPS,
RFID,
Hypervisor,
Web Servers,
Email, Messaging,
Clickstreams, Mobile,
Telephony, IVR, Databases,
Sensors, Telematics, Storage,
Servers, Security Devices, Desktops
3

4. Building a Big Data Platform
HA / DR Admin Data Security Apps SDKs/APIScale
Collect
Data
Index
Data
Enrich
Data
Search &
Explore
Analyze
& Predict
Report &
Visualize
Alert &
Action
4

5. Fully Integrated Enterprise Platform
HA / DR Admin Data Security Apps SDKs/APIScale
Collect
Data
Index
Data
Enrich
Data
Search &
Explore
Analyze
& Predict
Report &
Visualize
Alert &
Action
5

6. Structured
RDBMS
SQL Search
Schema at Write Schema at Read
Traditional Splunk
Splunk Approach to Machine Data
Copyright © 2014 Splunk Inc.
6
ETL Universal Indexing
Volume Velocity Variety
Unstructured

7. 7
Turning Machine Data Into Business Value
Index Untapped Data: Any Source, Type, Volume
Online
Services Web
Services
Servers
Security GPS
Location
Storage
Desktops
Networks
Packaged
Applications
Custom
ApplicationsMessaging
Telecoms
Online
Shopping
Cart
Web
Clickstreams
Databases
Energy
Meters
Call Detail
Records
Smartphones
and Devices
RFID
On-
Premises
Private
Cloud
Public
Cloud
Ask Any Question
Application Delivery
Security, Compliance
and Fraud
IT Operations
Business Analytics
Industrial Data and
the Internet of Things

8. 8
Splunk App for AWS
EC2
EMR
Kinesis
R53
VPC
ELB
S3
CloudFront
CloudTrail
CloudWatch
Redshift
SNS
API Gateway
Config
RDS
CF
IAM
Lambda
Explore Analyze Dashboard Alert Act
AWS Data Sources
End State: Comprehensive AWS Visibility

9. 9
True End State: Complete Hybrid Visibility
Index Untapped Data: Any Source, Type, Volume
Online
Services Web
Services
Servers
Security GPS
Location
Storage
Desktops
Networks
Messaging
Telecoms
Online
Shopping
Cart
Web
Clickstreams
Databases
Energy
Meters
RFID
On-
Premises
Private
Cloud
Public
Cloud
End-to-End Visibility
Application Delivery
Security, Compliance,
and Fraud
IT Operations
Business Analytics
Industrial Data and
the Internet of Things
CloudTrail
Config
Lambda
EC2

10. IT
Operations
Application
Delivery
Developer Platform (REST API, SDKs)
Business
Analytics
Industrial Data
and Internet of
Things
10
Delivers Value Across IT and the Business
Business
Analytics
Industrial Data
and Internet of
Things
Security,
Compliance
and Fraud

11. Platform for Application Delivery and IT Operations
ROOT CAUSE
AND ISSUE
RESOLUTION
PROACTIVE
MONITORING
AND REAL-TIME
ALERTING
DELIVER BETTER
QUALITY CODE
FASTER
CLOUD APP AND
INFRASTRUCTURE
MONITORING
MOBILE APP
TROUBLESHOOTING
USER & USAGE ANALYTICS
IT SERVICE INTELLIGENCE

12. Fault
Management
& Alerting
Infrastructure Monitoring App
Website and API Monitoring
Application Management Data Fabric
Ticket Analytics App
Machine Learning and Social Media
Sentiment
Web Analytics App
Capacity Management App
Copyright © 2016 Accenture. All rights reserved.

13. Better Code, Faster Development
and Migration to Cloud
• Reduced error rates by 2 orders of magnitude in a couple of weeks
• Rapidly found and fixed one line of code responsible for 30,000+ errors
• Real-time dashboards on error rates and production impact
• In-depth visibility as they strategically migrate apps to AWS Cloud

14. 14
Application Delivery & IT Ops Landscape
API
SDKs UI
Server, Storage,
Network
Server
Virtualization
Operating
Systems
Custom
Applications
Business
Applications
Cloud
Services
App Performance
MonitoringTicketing/Other
Web Intelligence
Mobile
Applications
Stream

15. 15
Single Platform for Security Intelligence
SECURITY &
COMPLIANCE
REPORTING
REAL-TIME
MONITORING OF
KNOWN THREATS
DETECT
UNKNOWN
THREATS
INCIDENT
INVESTIGATIONS
& FORENSICS
FRAUD
DETECTION
INSIDER
THREAT
Splunk Complements, Replaces and Goes Beyond Existing SIEMs

16. How FINRA Uses Splunk Cloud for Security
• Transforms third-party threat intelligence information into security alerts
• Leverages the Splunk App for AWS
• Efficient provisioning dramatically reduces costs
“Splunk Cloud gives you applications which let you get huge amounts of value from your data.”
— Sr. Director of Information Security

17. API
SDKs UI
Network Traffic
Analysis
Identity & Access
Control
Perimeter
Defense
EmailPayload Analysis
Endpoint Behavior
Analysis
Endpoint Change
Tracking
DLP
Security Analytics
Threat Intelligence
Cloud Security
Security & Compliance Landscape
17

18. 18
Extending Splunk for Business Analytics
Splunk Software Complements Existing BI Solutions
CUSTOMER
EXPERIENCE
PRODUCT
ANALYTICS
BUSINESS
PROCESS
ANALYTICS
DIGITAL
MARKETING

19. Why Domino’s uses Splunk for Application
Management and Business Analytics
• Understand device and app usage trends for orders
• Real-time reNex insights from store data
• Visibility into online and mobile coupon redemption
• Refine Campaigns for higher conversion

20. 20
Splunk for Industrial Data & the
Internet of Things
REMOTE
TROUBLESHOOTING
& PREVENTIVE
MAINTENANCE
SECURITY &
COMPLIANCE
DEVICE USAGE &
CUSTOMER
ANALYTICS
OPERATIONAL
EFFICIENCY

21. Splunk HEC, AWS Lambda, Kinesis & IoT
Lambda
Kinesis
IoT
Other
Services
HTTP Event Collector
(data queuing system)
(serverless code
execution)
Splunk/HEC Blueprints available on AWS Lambda Console
events > Kinesis > Lambda > Splunk Correlations/Alerts)

22. Saving Customers $Billions
on Fuel, Operations
• Improved customer operations by mining large volumes of unstructured data
• Moved from monthly batch analysis to flexible real-time reporting
• Delivered value-added services
• Minimized in-train forces
• Optimized operational efficiency
“Thanks to Splunk, our systems allow our customers to provide engineers with real-time feedback
and use operational insight to achieve optimal runs every time.”
— Director of Engineering, Train Dynamic Systems (a division of NYAB)

23. 23
All the features of Splunk Enterprise
All the benefits of SaaS

24. Hybrid
24
Search Head(s)
Indexer(s)
On Premises Private Cloud Public
Cloud
Search Head(s)
Indexer(s)
On Premises Private Cloud Public
Cloud
Hybrid Search
Single Pane of Glass Visibility

25. Platform for Operational Intelligence
The Splunk Portfolio
Rich Ecosystem of
Apps & Add-Ons
Splunk Premium
Solutions
Mainframe
Data
Relational
Databases
MobileForwarders Syslog/TCP
IoT
Devices
Network
Wire Data
Hadoop

26. Dev.splunk.com40,000+ questions
and answers
1,000+ apps Local User Groups
and
SplunkLive! events
26
Thriving Community

27. COLLECT DATA
FROM ANYWHERE
SEARCH
AND ANALYZE
EVERYTHING
GAIN REAL-TIME
OPERATIONAL
INTELLIGENCE
The Power of Splunk
27

28. FREE
CLOUD TRIAL
FREE
DOWNLOAD
FREE
AMAZON MACHINE
IMAGES (AMI)
28
Easy to Try & Get Started
1 32

29. Thank you

30. HOW SJS ARE USING SPLUNK>
& AWS TO DRIVE EFFICIENCIES
PHIL TANNER @Phil_Tanner
CHIEF TECHNICAL OFFICER
Student Job Search Aotearoa
June 2016
Splunk AWS v0.05.20160616.PT

31. AGENDA
Who am I?
Who is SJS?
Challenges and
needs
Use cases
www.sjs.co.nz @Phil_Tanner

32. WHO IS SJS?
Charity – owned by students
Last year, we filled 28,000
vacancies
Receive a call every 1.7 minutes
Earning our students over
NZ$80,000,000
With an average of $5,250 for
each student
Place a student every 5 minutes
www.sjs.co.nz @Phil_Tanner

33. What did we need from
our data solution?
What were our core
requirements?
Performance
Measurement
Understand
the data
Historical
Analytics
Increase
Efficiencies
Cross -
Departmental
Access
Minimal
CAPEX
www.sjs.co.nz @Phil_Tanner
CHALLENGES FACED BY SJS

34. www.sjs.co.nz @Phil_Tanner
How is the company performing?
Spot outliers
And trends
And relationships
Faster
To create
and to check
PERFORMANCE MEASUREMENT

35. www.sjs.co.nz @Phil_Tanner
GET THE REAL STORY
Why are Student Registrations
decreasing?
Overlaying Activated Registrations
provides a completely different story
Decreased time-wasters means it’s a good
news story, not a bad one.

36. Supply and demand in the marketplace
Provide historical context to current trends
www.sjs.co.nz @Phil_Tanner
Historical Analysis

37. www.sjs.co.nz @Phil_Tanner
INCREASED EFFICIENCIES
Helping the business run more smoothly

38. www.sjs.co.nz @Phil_Tanner
NOT JUST A TOOL FOR IT
Allows departments to
track their own progress
Easy to understand
Encourages friendly
competition

39. www.sjs.co.nz @Phil_Tanner
AVOIDING CAPEX
Better cash-flow
Higher uptime
Faster/cheaper upgrades

40. www.sjs.co.nz @Phil_Tanner
CONCLUSION
Performance measurement
Measuring KPIs
Improving data integrity
Get the Real Story
Understand the story behind the
numbers
Security
Track brute force attacks in real time
Identify website security breaches
Single-Sign-On to access reports
Systems Monitoring
Perfect for log file analysis
Proactive problem identification
Post-analysis
Increased Efficiencies
Helping the business run more smoothly
Cross-Departmental Access
Encourages friendly competition
Departments track their
own progress
Easy to understand
Reduction in CAPEX
Better cash-flow
Higher uptime
Faster/cheaper upgrades

41. Phil.Tanner@SJS.co.nz
ChiefTechnicalOfficer
StudentJobSearchAotearoa
www.sjs.co.nz @Phil_Tanner
THANK YOU FOR LISTENING
During this talk, SJS has placed six students in work,
generating $14,375 for the New Zealand economy

43. MIX DATASOURCES INTO ONE NUMBER
Pull our audited numbers
from the Splunk index
Extract data up to last night
from the reporting database
Grab today’s data from CSV
index=auditedplacementreports
| where _time >= relative_time(now(), "-6mon@y+6mon")
| eval class = "audited"
| stats count as placement by class
| append [ | dbquery sjs_reporting [ | stats count
| eval last_audited_date = round([ search index=audited_placement_reports | stats max(_time) as x | return $x],0)
| eval sql_str = " "
SELECT
'unaudited' AS `class`,
COUNT(`a`.`id`) AS `placement`
FROM
`applications` `a`
WHERE
`a`.`status` = 'accepted'
AND `a`.`confirmed_date` > FROM_UNIXTIME(".last_audited_date.")
" "
| return $sql_str ]
| appendcols [ | inputlookup todaysplacements
| stats count as today_placement ]
| eval placement = placement+today_placement
| fields class placement ]
| transpose header_field=class
| eval achieved = toString( audited+unaudited, "commas")
| fields achieved
www.sjs.co.nz @Phil_Tanner

44. Copyright © 2015 Splunk Inc.
Splunk App for AWS

45. Agenda
• This session is meant for AWS users who are setting up the Splunk App for
AWS (or people who will be doing so after this session!).
• Using the Splunk App for AWS you will gain visibility across your AWS
Deployment.

46. Agenda
• AWS Components
• Permissions for SQS, SNS and S3 buckets
• Setting up Splunk App for AWS
• Questions & Answers – at the Splunk booth after this session!

47. Splunk Offerings in AWS
• Splunk App for
AWS: Integrates
w/CloudTrail,
Config and Billing,
VPC Flow Logs
Integrations• Self-managed cloud
deployments
• Self-deploy in AWS
• Integrated with EMR
• Search data in S3
• Hourly pricing
Self-managed
• Cloud service
designed for small IT
environments
• $90 a month
• Splunk Enterprise
as a service
• Full app, SDK, API,
platform support
Cloud-service

48. AWS Architecture Diagram
Amazon Instances
Amazon Logging Layer
Amazon Messaging
Amazon Storage / Queues
Splunk Collects the data
from the AWS SQS and
the S3 bucket using the
AWS SDK for python
(Boto3).

49. Requirements For Splunk App For AWS
• Splunk
• Splunk 6.1 or later
• Splunk Add-on for Amazon
Web Services
• Splunk Add-on for Amazon
Web Services +1.1.0
required for AWS Config
• AWS
• AWS CloudTrail: Enable CloudTrail
with SQS and SNS.
• AWS Config: Enable Config with
SQS and SNS.
• Billing: Refer to the AWS
documentation to turn on AWS
detailed billing.
• VPC Flow Logs: Enable VPC Flow
log collection.

50. Install the Splunk Add-on for AWS
1. Configure your AWS accounts and services, or confirm your existing
configurations.
2. Configure your AWS account permissions to match those required
by the add-on.
3. Install the add-on.
4. Set up the add-on on your forwarders or single instance.
5. Configure your inputs to get your AWS data into Splunk Enterprise.
6. This is all very well documented at docs.splunk.com

51. Permissions
S3 Storage
CloudTrail
API Tracking
SNS Notification
SQS Message Queue
splunkuser
Sample permissions for cloudtrail

52. Splunk Architecture
• Distributed Splunk
Deployment
Single Splunk Deployment
Splunk'server
Indexer
Heavy+Weight+
Forwarder
Splunk+search
IndexerIndexer
Splunk Add-
on for AWS
installed on
Heavy Weight
Forwarder
Splunk App
for AWS
installed on
all-in-one
Splunk server

53. Setup Interface
Add Your
Account
Add your
AWS
Inputs

54. Wait 5 – 10 Minutes
• Yes, you’ll need to wait before all the dashboards and reports populate.

55. Gain Visibility Into AWS Logs

57. THANKYOU