ChatOps 
… 
a 
beginner’s 
guide 
~ 
$
Jason 
Hand 
VictorOps 
jason@victorops.com 
@jasonhand
Agenda 
• Introduc)on 
to 
ChatOps 
• Origins 
• Tools 
• Benefits 
• Examples 
• Limita)ons 
& 
Concerns 
• Ge>ng 
Started 
• Q&A
What 
do 
you 
do 
over 
and 
over 
again? 
Can 
I 
do 
that 
for 
you? 
I’ll 
think 
about 
it. 
@jasonhand 
#ChatOps
ChatOps 
“puCng 
tools 
in 
the 
middle 
of 
the 
conversaFon” 
-­‐ 
GitHub 
(2010) 
“everyone 
is 
pairing 
all 
of 
the 
Fme” 
-­‐ 
Jesse 
Newland 
(Ops 
– 
GitHub)
DevOps 
Culture 
AutomaFon 
Measurement 
Sharing 
To 
move 
fast 
and 
maintain 
stability 
it’s 
important 
to 
have 
a 
culture 
of 
automaFon, 
measurement 
and 
sharing 
@jasonhand 
#ChatOps
Tools 
• Chat 
clients 
• (with 
channels 
or 
rooms) 
• Bots 
• Scripts 
@jasonhand 
#ChatOps
HipChat 
Campfire 
Chat 
Slack 
@jasonhand 
#ChatOps
Bots 
@jasonhand 
#ChatOps
Scripts 
@jasonhand 
#ChatOps
Benefits 
• Sharing 
• Learning 
• Speed 
@jasonhand 
#ChatOps
MulF-­‐user 
terminal 
Share 
learning 
experiences 
Sharing 
Disseminate 
knowledge 
across 
everyone 
on 
the 
team 
Share 
learning 
experiences 
@jasonhand 
#ChatOps
Learning 
@jasonhand 
#ChatOps
Speed 
How 
many 
clicks 
does 
it 
take 
you?
Jason: 
Hey 
Dan 
… 
did 
you 
see 
that 
spike 
in 
CPU 
usage 
on 
Prod1.. 
~ 
3 
minutes 
ago? 
Dan: 
No. 
Let 
me 
check. 
brb 
Old 
methods 
Dan: 
Shit. 
Yeah, 
I 
see 
it. 
Looks 
ok 
now. 
I 
think 
:-­‐/ 
Jason: 
Think 
we 
should 
let 
Mike 
know? 
He 
might 
want 
to 
dig 
in 
to 
it 
later. 
Dan: 
Yeah, 
I’ll 
take 
a 
screenshot 
and 
email 
it 
to 
him. 
Jason: 
Would 
anyone 
else 
need 
to 
know 
about 
it? 
Dan: 
Prob. 
I’ll 
cc 
the 
DevOps 
team 
so 
everyone 
knows
Jason: 
Hey 
Dan. 
My 
code 
is 
ready 
to 
push 
when 
you 
are. 
Dan: 
Old 
methods 
Cool. 
I’ll 
commit 
it 
to 
the 
latest 
branch 
here 
in 
a 
minute. 
Jason: 
Great. 
Also, 
we 
should 
probably 
sit 
down 
with 
Nick 
(new 
guy) 
and 
show 
him 
how 
to 
do 
that 
soon. 
Dan: 
Yeah. 
Once 
I 
get 
some 
Fme, 
I’ll 
schedule 
something. 
:/ 
Jason: 
There’s 
probably 
a 
bunch 
more 
stuff 
to 
show 
him 
too, 
but 
I’m 
not 
sure 
I 
even 
know 
all 
of 
the 
steps 
for 
everything. 
Dan: 
We’ve 
got 
a 
wiki 
for 
most 
things, 
but 
not 
everything. 
I’ll 
see 
what 
I 
can 
find 
and 
email 
him.
Jason: 
Hey 
Dan. 
My 
code 
is 
ready 
to 
push 
when 
you 
are. 
Dan: 
Cool. 
I’ll 
commit 
it 
to 
the 
latest 
branch 
here 
in 
a 
minute. 
Jason: 
Great. 
Also, 
we 
should 
probably 
sit 
down 
with 
Nick 
(new 
guy) 
and 
show 
him 
how 
to 
do 
that 
soon. 
Dan: 
Yeah. 
Once 
I 
get 
some 
Fme, 
I’ll 
schedule 
something. 
:/ 
Jason: 
There’s 
probably 
a 
bunch 
more 
stuff 
to 
show 
him 
too, 
but 
I’m 
not 
sure 
I 
even 
know 
all 
of 
the 
steps 
for 
everything. 
Dan: 
We’ve 
got 
a 
wiki 
for 
most 
things, 
but 
not 
everything. 
I’ll 
see 
what 
I 
can 
find 
and 
email 
him. 
Old 
methods 
Chat 
with 
Jason 
& 
Dan 
If 
using 
a 
tradiFonal 
chat 
client 
Such 
as 
Gchat 
… 
Who 
else 
can 
see 
this 
conversaFon? 
Nobody 
! 
@jasonhand 
#ChatOps
What 
do 
you 
do 
over 
and 
over 
again? 
Can 
I 
do 
that 
for 
you? 
Heck 
yeah! 
:D 
@jasonhand 
#ChatOps
Examples 
• 
Run 
MySQL 
command 
(as 
read-­‐only 
user) 
• Authorize 
users 
(assign 
and 
enforce 
user-­‐based 
roles) 
• Take 
a 
server 
offline 
• Bring 
a 
server 
online 
• Deploy 
code 
to 
a 
repository 
• Displaying 
the 
status 
of 
a 
JIRA 
Fcket 
• Adding 
comments 
and 
changing 
their 
status 
on 
Fckets 
• Query 
the 
status 
of 
API's 
(internal 
& 
external) 
• VM 
interacFon 
to 
see 
host 
health 
• Monitoring 
service 
interacFon 
to 
trigger 
status 
checks 
• Hubot 
scripts 
can 
be 
wrinen 
in 
CoffeeScript 
or 
Javascript 
• NoFfy 
people 
not 
present 
in 
a 
chatroom 
that 
they 
were 
menFoned 
• Fetch 
logs 
• Rebalance 
shards 
across 
its 
file 
servers 
• List 
currently 
running 
jobs/processes 
• Kill 
jobs/processes 
• Tail 
a 
job/process 
• Track 
Customer 
signups 
• Sharing 
graphs 
for 
context 
• Store 
staFc 
links, 
mapping 
those 
links 
to 
simple, 
easy 
to 
remember 
commands 
In 
GitHub 
Opening 
issues 
Closing 
issues 
CommenFng 
on 
issues 
LisFng 
issues 
Merge 
pull 
requests 
jhand.ml/scripts 
jhand.ml/heroku 
@jasonhand 
#ChatOps
Example 
script 
Show 
current 
GitHub 
status 
and 
messages 
jhand.ml/script1 
@jasonhand 
#ChatOps
Jason: 
hubot 
github 
status 
hubot 
github 
status 
-­‐ 
Returns 
the 
current 
system 
status 
and 
Fmestamp. 
hubot 
github 
status 
last 
-­‐ 
Returns 
the 
last 
human 
communicaFon, 
status, 
and 
Fmestamp. 
hubot 
github 
status 
messages 
-­‐ 
Returns 
the 
most 
recent 
human 
communicaFons 
with 
status 
and 
Fmestamp. 
Hubot: 
github 
status: 
new 
file: 
Chatops.txt 
116 
seconds 
ago 
@jasonhand 
#ChatOps
Jason: 
Hey 
Dan. 
My 
code 
is 
ready 
to 
push 
when 
you 
are. 
Dan: 
Cool. 
I’ll 
commit 
it 
to 
the 
latest 
branch 
here 
in 
a 
minute. 
Jason: 
Great. 
Also, 
we 
should 
probably 
sit 
down 
with 
Nick 
(new 
guy) 
and 
show 
him 
how 
to 
do 
that 
soon. 
Dan: 
Yeah. 
Once 
I 
get 
some 
Fme, 
I’ll 
schedule 
something. 
:/ 
Jason: 
There’s 
probably 
a 
bunch 
more 
stuff 
to 
show 
him 
too, 
but 
I’m 
not 
sure 
I 
even 
know 
all 
of 
the 
steps 
for 
everything. 
Dan: 
We’ve 
got 
a 
wiki 
for 
most 
things, 
but 
not 
everything. 
I’ll 
see 
what 
I 
can 
find 
and 
email 
him. 
New 
and 
improved! 
Chat 
Room 
Who 
else 
can 
see 
this 
conversaFon? 
Erybody 
! 
Members 
Jason 
Dan 
Nick 
Mike 
Jeff 
Doug 
Tara 
@jasonhand 
#ChatOps
Jason: 
Hubot 
graphite 
load 
1 
min 
on 
serverABC 
Jason: 
@Dan 
… 
you 
seeing 
this? 
Dan: 
Yeah. 
New 
method 
w/ 
bots 
Hubot: 
Mike: 
That 
was 
me. 
I 
was 
running 
a 
giant 
report. 
Should 
be 
fine 
now 
Nick 
(new 
guy): 
Cool. 
Thanks 
for 
showing 
me 
how 
to 
do 
that. 
;-­‐)
Big 
Whoop 
Why 
is 
this 
important?
Graphs 
or 
it 
didn’t 
happen 
@jasonhand 
#ChatOps
@ngrieb: 
hubot 
graphite 
load 
1 
min 
on 
serverABC 
@jasonhand 
#ChatOps
@jasonhand 
#ChatOps
GeCng 
Started 
Install 
bot 
jhand.ml/installhubot 
Create 
repo 
Download 
exisFng 
scripts 
jhand.ml/scripts 
Create 
your 
own 
scripts 
Ask 
others 
what 
needs 
to 
be 
automated 
@jasonhand 
#ChatOps
Concerns
LimitaFons 
@jasonhand 
#ChatOps
What 
I’d 
like 
to 
learn 
from 
you 
: 
Are 
you 
doing 
ChatOps? 
Are 
you 
on-­‐call? 
jhand.ml/SOoncall 
@jasonhand 
#ChatOps
Automate 
! 
@jasonhand 
#ChatOps
Jason 
Hand 
VictorOps 
jason@victorops.com 
@jasonhand

Chat ops .. a beginner's guide

  • 1.
    ChatOps … a beginner’s guide ~ $
  • 2.
    Jason Hand VictorOps jason@victorops.com @jasonhand
  • 3.
    Agenda • Introduc)on to ChatOps • Origins • Tools • Benefits • Examples • Limita)ons & Concerns • Ge>ng Started • Q&A
  • 4.
    What do you do over and over again? Can I do that for you? I’ll think about it. @jasonhand #ChatOps
  • 5.
    ChatOps “puCng tools in the middle of the conversaFon” -­‐ GitHub (2010) “everyone is pairing all of the Fme” -­‐ Jesse Newland (Ops – GitHub)
  • 6.
    DevOps Culture AutomaFon Measurement Sharing To move fast and maintain stability it’s important to have a culture of automaFon, measurement and sharing @jasonhand #ChatOps
  • 7.
    Tools • Chat clients • (with channels or rooms) • Bots • Scripts @jasonhand #ChatOps
  • 8.
    HipChat Campfire Chat Slack @jasonhand #ChatOps
  • 9.
  • 11.
  • 12.
    Benefits • Sharing • Learning • Speed @jasonhand #ChatOps
  • 13.
    MulF-­‐user terminal Share learning experiences Sharing Disseminate knowledge across everyone on the team Share learning experiences @jasonhand #ChatOps
  • 14.
  • 15.
    Speed How many clicks does it take you?
  • 16.
    Jason: Hey Dan … did you see that spike in CPU usage on Prod1.. ~ 3 minutes ago? Dan: No. Let me check. brb Old methods Dan: Shit. Yeah, I see it. Looks ok now. I think :-­‐/ Jason: Think we should let Mike know? He might want to dig in to it later. Dan: Yeah, I’ll take a screenshot and email it to him. Jason: Would anyone else need to know about it? Dan: Prob. I’ll cc the DevOps team so everyone knows
  • 17.
    Jason: Hey Dan. My code is ready to push when you are. Dan: Old methods Cool. I’ll commit it to the latest branch here in a minute. Jason: Great. Also, we should probably sit down with Nick (new guy) and show him how to do that soon. Dan: Yeah. Once I get some Fme, I’ll schedule something. :/ Jason: There’s probably a bunch more stuff to show him too, but I’m not sure I even know all of the steps for everything. Dan: We’ve got a wiki for most things, but not everything. I’ll see what I can find and email him.
  • 18.
    Jason: Hey Dan. My code is ready to push when you are. Dan: Cool. I’ll commit it to the latest branch here in a minute. Jason: Great. Also, we should probably sit down with Nick (new guy) and show him how to do that soon. Dan: Yeah. Once I get some Fme, I’ll schedule something. :/ Jason: There’s probably a bunch more stuff to show him too, but I’m not sure I even know all of the steps for everything. Dan: We’ve got a wiki for most things, but not everything. I’ll see what I can find and email him. Old methods Chat with Jason & Dan If using a tradiFonal chat client Such as Gchat … Who else can see this conversaFon? Nobody ! @jasonhand #ChatOps
  • 19.
    What do you do over and over again? Can I do that for you? Heck yeah! :D @jasonhand #ChatOps
  • 20.
    Examples • Run MySQL command (as read-­‐only user) • Authorize users (assign and enforce user-­‐based roles) • Take a server offline • Bring a server online • Deploy code to a repository • Displaying the status of a JIRA Fcket • Adding comments and changing their status on Fckets • Query the status of API's (internal & external) • VM interacFon to see host health • Monitoring service interacFon to trigger status checks • Hubot scripts can be wrinen in CoffeeScript or Javascript • NoFfy people not present in a chatroom that they were menFoned • Fetch logs • Rebalance shards across its file servers • List currently running jobs/processes • Kill jobs/processes • Tail a job/process • Track Customer signups • Sharing graphs for context • Store staFc links, mapping those links to simple, easy to remember commands In GitHub Opening issues Closing issues CommenFng on issues LisFng issues Merge pull requests jhand.ml/scripts jhand.ml/heroku @jasonhand #ChatOps
  • 21.
    Example script Show current GitHub status and messages jhand.ml/script1 @jasonhand #ChatOps
  • 22.
    Jason: hubot github status hubot github status -­‐ Returns the current system status and Fmestamp. hubot github status last -­‐ Returns the last human communicaFon, status, and Fmestamp. hubot github status messages -­‐ Returns the most recent human communicaFons with status and Fmestamp. Hubot: github status: new file: Chatops.txt 116 seconds ago @jasonhand #ChatOps
  • 23.
    Jason: Hey Dan. My code is ready to push when you are. Dan: Cool. I’ll commit it to the latest branch here in a minute. Jason: Great. Also, we should probably sit down with Nick (new guy) and show him how to do that soon. Dan: Yeah. Once I get some Fme, I’ll schedule something. :/ Jason: There’s probably a bunch more stuff to show him too, but I’m not sure I even know all of the steps for everything. Dan: We’ve got a wiki for most things, but not everything. I’ll see what I can find and email him. New and improved! Chat Room Who else can see this conversaFon? Erybody ! Members Jason Dan Nick Mike Jeff Doug Tara @jasonhand #ChatOps
  • 24.
    Jason: Hubot graphite load 1 min on serverABC Jason: @Dan … you seeing this? Dan: Yeah. New method w/ bots Hubot: Mike: That was me. I was running a giant report. Should be fine now Nick (new guy): Cool. Thanks for showing me how to do that. ;-­‐)
  • 25.
    Big Whoop Why is this important?
  • 26.
    Graphs or it didn’t happen @jasonhand #ChatOps
  • 27.
    @ngrieb: hubot graphite load 1 min on serverABC @jasonhand #ChatOps
  • 28.
  • 29.
    GeCng Started Install bot jhand.ml/installhubot Create repo Download exisFng scripts jhand.ml/scripts Create your own scripts Ask others what needs to be automated @jasonhand #ChatOps
  • 30.
  • 31.
  • 32.
    What I’d like to learn from you : Are you doing ChatOps? Are you on-­‐call? jhand.ml/SOoncall @jasonhand #ChatOps
  • 33.
  • 34.
    Jason Hand VictorOps jason@victorops.com @jasonhand