Splunk at Statnett

1. Copyright
©
2015
Splunk
Inc.
Linus Myrefelt @ Statnett SF
The
Future
is
Electric

2. 2
The
future
is
"Electric"

3. 3
Important
What
I
am
going
to
talk
about
does
not
neccessarily
represent
:
• The
truth
• Splunk’s
opinions
or
thoughts
• StatneG’s
opinions
or
thoughts
• My
own
thoughts

4. 4
Agenda
" StatneG?
" Linux
" Where
it
all
started
-­‐
"SPll"
troubleshooPng
(/
Root
cause
analyPcs)
" The
new
driver
-­‐
devOps
/
agile
development
and
rapid
deployment
" What
we
want
to
do
-­‐
ApplicaPon
management
/
IT
Service
Management
" What
we
all
do
-­‐
Security
(doh!)
" The
future
is
electric!
–
Next
step
on
our
journey
" Take
aways
and
Pps
for
your
journey
and
success!

5. 5
StatneG,
what?
" Make
sure
that
the
lights
are
on
in
Norway
" State
owned
company
" Quite
small
(~1500)
" We
own,
build
and
maintain
the
Norwegian
power-­‐grid
" Regulates
the
market
" "The
spider
in
the
web"

6. 6
SPll
a
young
company
" Light-­‐weight
company
" Small
environment
in
server/endpoint
numbers
" Large
and
complex
network
" MS
dominated
" Large
group
of
developers
" Heavily
project
focused
organisaPon
" Heavily
depending
on
IT
" Heavily
regulated

7. 7
(linu(s|x))
Background
and
Role
" Born
and
raised
in
the
smålandian
woods
" Geek
/
"Hacker"
since
childhood
" Living
Oslo
/
Norway
(same
same
but
different)
" Trying
to
speak
Swedish
in
Norway
and
Norwegian
in
Sweden
" Splunker
since
~version
4
" Before:
Consultant
doing
APM,
NPM,
Splunk
and
"security"
" Now:
Building
"Next-­‐Gen"
log
and
monitoring
plagorm
at
StatneG
" Not
a
"PowerPoint
warrior"

8. 8
My
3
(4
including
Splunk
t-­‐shirts)
favorite
things
J

9. 9
This
is
what
I
belive
in!

10. 10
This
is
what
I
belive
in!

11. 11

12. 12
Let’s
get
down
to
business
–
use
cases
TroubleshooPng
Development
IT
Service
Management
Security

13. 13
Where
it
all
started
-­‐
troubleshooPng
(/
Root
cause
analyPcs)

14. 14
Our
iniPal
pain
How
do
you
troubleshoot
amongst
1000s
of
servers?
What
about
many
1000s
of
network
devices?
What
if
you
have
100s
of
thousands
of
communicaPon
points?
How
do
you
go
about
and
do
just
that?

15. 15
Our
SoluPon
to
the
problem
…
started
with
networking

16. 16
Enabled
us
to…
" Maintain
our
infrastructure
posture
" Track
faulty
devices
" Earlier
and
controlled
replacement
" Correlate
events
" Spot
trends
on
network
" Bigger
picture
with
drilldown

17. 17
And
the
SituaPon
now?
" The
good
guys
use
Splunk
for
root
cause
analy5cs
(tuff
word)
" The
bad
ones
use
me
or
my
colleague
for
root
cause
analy5cs
(s5ll
a
tuff
word)

18. 18
The
new
driver
-­‐
devOps
/
Agile
development
and
rapid
deployment

19. 19
Our
developers
were
struggling
with:
" MulPple
Stages
" Across
"zones
/
network
segments"
" Amongst
mulPple
servers
" Use
of
crypPc
tool
with
a
hard
to
get
syntax
–
tail,
grep,awk,sed
mm
" Customized
event
viewer
" Not
scalable
" Genng
access
to
the
right
data
" In
a
Pmely
fashion

20. 20
SoluPon
-­‐
They
threw
it
into
splunk
J
And
they
created
a
big
fat
mess
-­‐
…
sPll
like
using
grep
and
awk
for
your
life

21. 21
What
we
want
to
do
-­‐
ApplicaPon
management
/
IT
Service
management

22. 22
Our
ops
guys
were
struggling
with…
" SPll
kind
of
of
young
company
" Started
to
mature
" Old
but
good
siloed
tools
" Not
very
user-­‐friendly
or
accessible
" Need
something
more
unifying
" HolisPc
overview
of
services
and
KPIs
" Give
stak´holders
the
right
informaPon
" Technical
overview
with
drill
downs
into
alerts
and
events

23. 23
Our
soluPon
for
Ops
" Re-­‐designed,
re-­‐architectured
and
scaled
up
soluPon
" Splunk
agent
deployed
" Part
of
standard
image
and
rouPnes
" Different
departments
pushing
for
expansion
" Need
to
seGle
on
informaPon
model

24. 24
APM
/
IT
Service
Management

25. 25
Security
–
All
"your"
data
belongs
to
me

26. 26
In
Security
we
struggle
with
the
following
things
" Too
few
people
…
already
heavily
occupied
" Not
enough
(good)
people
to
hire
" No
single
pane
of
overview
" Hard
to
keep
up
with
todays
threat
" No
real
"Malware
popup"

27. 27
We
want
to
do
more
" Improve
our
security
posture
" Enable
the
right
peope
with
data
" Do
more
with
less
" AND
" Being
able
to
keeping
track
of
aGackers
" Threat
intel
,
i.e
blacklists
…
=
Noise
" Researching
IP
/
AGackers
is
part
of
the
game

28. 28
How
we
are
trying
to
do
it
" UPlizing
Splunk
and
data
as
enabler
" Automate
boring
and
Pme-­‐consuming
tasks
" We
combine
freely
tools
with
homebrewed
" Scraping
public
api
and
web
services
" Everything
"hosPle"
that
goes
in
and
out

29. 29
Security
Manager
approved

30. 30
The
future
is
electric!
–
Next
step
on
our
Journey

31. 31
The
future
is
electric!
" ConPnue
to
roll
out
agent
" Collect
applicaPon
logs
" Expanding
use-­‐cases
" Work
hard
on
normalisaPon
" InformaPon
model
" Service
modelling
" More
integraPons
into
splunk
" Keep
adding
reports
and
alerts

32. 32
Top
Takeaways
/
My
Tips
" Invest
in
educaPon
for
(different)
users
" Use
PS
or
a
trusted
local
partner
" Before
reaching
maturity
…
maybe
start
small

33. 33
Quote
Box
Our
mission
is
to
make
machine
data
accessible,
useable
and
valuable
to
everyone.

34. Thank
You