Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Effec%ve'Devops:
Collabora'on)and)Tools
Jennifer'Davis'&'Katherine'Daniels
✨ @beerops(@sigje(#effec/vedevops✨ 1
Jennifer'Davis
• So$ware)Engineer,)Chef)So$ware
• Co2author)of)"Effec8ve)Devops"
• Founder)of)CoffeeOps
• DevOpsDays)Silicon...
Katherine)Daniels
• Senior(Opera,ons(Engineer,(Etsy
• Co4author(of("Effec,ve(Devops"
• DevopsDays(NYC(Organizer
• Ship(Show...
Communica)on
• Jennifer(Davis
Twi/er:(@sigje
Email:(sparklydevops@gmail.com
• Katherine(Daniels
Twi/er:(@beerops
Email:(sp...
Feedback
• Construc*ve-feedback
• What-did-you-find-helpful?
• What-would-you-like-to-see-more/less-of?
• Was-there-anythin...
Schedule
• Introduc*on+to+teams,+devops+principles
• 10:30911:00am+Morning+Break+9+Su?on+Complex
• Visualiza*on+of+work,+G...
Schedule
• Tes%ng(infrastucture(automa%on(and(other(changes
• 3:0073:30pm(A:ernoon(Break(7(Su>on(Complex
• Measuring,(moni...
Network(Connec,vity
Network(Name:(Velocity
Access(Code:((none)
Local&Downloads:&h.p://10.10.32.101
✨ @beerops(@sigje(#effec...
Expecta(ons
• Safe&space&to&share&experiences,&learn&from&each&other
• Code&of&Conduct
• Learn&effec9ve&workflows&for&using&...
Team%Introduc.ons
• Meet%your%team!
• What%are%mo/va/ons?%What%are%current%beliefs?%What%are%
current%skills?%Gaps%in%skil...
What%is%Devops
✨ @beerops(@sigje(#effec/vedevops✨ 11
What%is%Devops
Cultural'movement'that'seeks'to:
• change(how(individuals(work,
• value(the(diversity(of(work(done,
• devel...
Folk%Models
• general(popularly(understood(meaning(par2cular(to(a(socio4
cultural(grouping(but(which(has(not(been(formally...
Why$Devops?
✨ @beerops(@sigje(#effec/vedevops✨ 14
High%Performing%Devops%Teams
are$more$agile
30X$more$frequent$deployments
8000X$faster$lead$-mes$than$peers
2014%PuppetLab...
High%Performing%Devops%Teams
are$more$reliable
2X#change#success#rate
12X$faster$mean$-me$to$recovery$(MTTR)
2014%PuppetLa...
Five%Pillars%of%Effec0ve%Devops
• Collabora(on
• Hiring
• Affinity
• Tools
• Scaling
✨ @beerops(@sigje(#effec/vedevops✨ 17
Collabora'on
• Individuals+Working+Together
✨ @beerops(@sigje(#effec/vedevops✨ 18
Hiring
• Choosing)Individuals
✨ @beerops(@sigje(#effec/vedevops✨ 19
Affinity
• From&Individuals&to&Teams
✨ @beerops(@sigje(#effec/vedevops✨ 20
Tools
• Accelerators+of+Culture
✨ @beerops(@sigje(#effec/vedevops✨ 21
Scaling
• Evolu'on,*growth*and*advancement*of*the*organiza'on
✨ @beerops(@sigje(#effec/vedevops✨ 22
Collabora'on)and)Tools
✨ @beerops(@sigje(#effec/vedevops✨ 23
Recognizing)your)Devops)Narra3ve
✨ @beerops(@sigje(#effec/vedevops✨ 24
The$Devops$Compact
• shared(mutual(understanding
• established(boundaries
✨ @beerops(@sigje(#effec/vedevops✨ 25
Team
• Common&purpose
• Defined&beliefs
• Empowered
✨ @beerops(@sigje(#effec/vedevops✨ 26
Careless'Conversa+ons'(inspired'by'Alan'Cyment)
• Pair&up
• Select&one&person&to&go&first.
• For&one&minute,&speaker&talks&...
Careless'Conversa+ons
• How%does%it%feel%not%to%be%listened%to?
• How%does%it%feel%to%ignore%someone?
✨ @beerops(@sigje(#e...
Cul$va$ng)Empathy
• Collect(stories
• Listen
• Circle(back
✨ @beerops(@sigje(#effec/vedevops✨ 29
Smarter'Teams'Build'Be/er'Value1
• Lots&of&Communica/on
• Contribute&equally&to&team's&discussions
• Theory&of&Mind
• Incr...
A"life"becomes"meaningful"when"one"sees"himself"or"herself"as"an"
actor"within"the"context"of"a"story.
!!"George"Howard
✨ ...
Small%vs%Large%teams
• Large'teams'+'roles'may'be'highly'segregated
• Small'teams'+'one'person'may'be'responsible'for'many...
Cri$cal(Habits(for(Teams
• Code&Review
• Pairing
✨ @beerops(@sigje(#effec/vedevops✨ 33
Code%Review
• Max%90%minutes%in%one%se0ng
✨ @beerops(@sigje(#effec/vedevops✨ 34
Pairing
• Agile'so*ware'development
• 2'people'work'together'on'1'worksta8on
• Driver':'writes'code
• Observer':'reviews'e...
Types&of&Pairing
• Expert(expert
• Expert(novice
• Novice(novice
✨ @beerops(@sigje(#effec/vedevops✨ 36
Visualiza(on+of+Work
• Bug/issue)queue
• Kanban
✨ @beerops(@sigje(#effec/vedevops✨ 37
Factors(for(Success
• Management)buy)in
• Training
• Minimize)'pushing'
• Team)sync
✨ @beerops(@sigje(#effec/vedevops✨ 38
Workflow
Work%that%is
• orchestrated
• repeatable
• organized
• moves4from4one4state4to4another
✨ @beerops(@sigje(#effec/ved...
WIP
• Work&in&Progress
• work&that&has&had&money&or&people&applied&to&it.
✨ @beerops(@sigje(#effec/vedevops✨ 40
Work%Iden*fica*on
• name
• start)date
• end)date
• current)state
• descrip/on
• priority
✨ @beerops(@sigje(#effec/vedevops✨ ...
Task%Handling
• What&is&it?
• Can&you&do&anything&with&it?
• What&is&the&next&step?
✨ @beerops(@sigje(#effec/vedevops✨ 42
Projects
• Same&requirements&as&a&task
• Larger&in&scope
• May&be&comprised&of&more&than&one&task
✨ @beerops(@sigje(#effec/...
Interrupts
• Non%planned%work%that%comes%in
• customer%request
• incident
• request%for%help%from%coworker
• single%point%...
Blocked(work
• Work&that&can&progress&no&further:
• dependent&teams&5&blocked&by&external&team
• insufficiently&qualified&req...
Team%Exercise
Discuss&with&your&team:
• What&is&the&difference&between&a&task&and&a&project?
• Do&you&have&interrupts?&What...
Kanban
• Start&with&what&you&do&now
• Agree&to&incremental,&evolu7onary&change
• Respect
• Everyone&is&a&leader
✨ @beerops...
Kanban%Prac)ces
• Visualize
• Limit-WIP
• Manage-flow
• Make-policies-explicit
• Implement-feedback-loops
✨ @beerops(@sigje...
Visualize
• Intent
• Alignment
• Coherence
✨ @beerops(@sigje(#effec/vedevops✨ 49
Limit%WIP
• Pull%(don't%push)
✨ @beerops(@sigje(#effec/vedevops✨ 50
Manage&Flow
• Monitor/measure/report
• Incremental2change
✨ @beerops(@sigje(#effec/vedevops✨ 51
Make%Policies%Explicit
• Document*processes
• Group*signoff
✨ @beerops(@sigje(#effec/vedevops✨ 52
Implement(Feedback(Loops
• Collabora(on
• Retrospec(ves
✨ @beerops(@sigje(#effec/vedevops✨ 53
Devops'Tools
• Establish*local*development*environment
• Version*control
• Manual*7>*Automa:on*7>*Con:nuous
• Ar:facts
• I...
Local&Development&Environment&(LDE)
• Consistent)set)of)tools)across)the)team
• Ability)to)quickly)onboard)new)engineers
✨...
Provisioned*Node*,*LDE
• AWS%instance%node
• Chef%DK
• Test%Kitchen
• Ruby
• ChefSpec,%ServerSpec
• Git
✨ @beerops(@sigje(...
Configura)on*Management
• Process(of(iden-fying,(managing,(monitoring,(and(audi-ng(a(
product(through(its(en-re(life(includ...
Version(Control
• Records)changes)to)files)or)sets)of)files)stored)within)the)system
• Enable)revisions
• Integrity)checking...
Ar#fact(Repository
• Secure
• Trusted
• Stable
• Accessible
• Versioned
✨ @beerops(@sigje(#effec/vedevops✨ 59
Introduc)on*to*Lab*1
Lab$1
User:&chef
Password:&chef
✨ @beerops(@sigje(#effec/vedevops✨ 60
Lab$1
Lab$1
Time:&15&minutes
✨ @beerops(@sigje(#effec/vedevops✨ 61
Introduc)on*to*Lab*2
Lab$2
✨ @beerops(@sigje(#effec/vedevops✨ 62
Lab$2
Lab$2
Time:&15&minutes
✨ @beerops(@sigje(#effec/vedevops✨ 63
Welcome'Back
• git%review%
✨ @beerops(@sigje(#effec/vedevops✨ 64
git$pull
git$pull$REMOTE
✨ @beerops(@sigje(#effec/vedevops✨ 65
git$push$remote$branch
• transfer)commits)from)a)local)repo)to)a)remote)repo.
• counterpart)to)git)fetch
✨ @beerops(@sigje...
Infrastructure
• Aggregate(of(applica/ons,(configura/ons,(access(control,(data,(
compute(nodes,(network,(storage,(processes...
Infrastructure+Automa/on
• Systems(that(reduce(the(burden(on(people(to(manage(services(
and(increase(the(quality,(accuracy...
Infrastructure+Automa/on+Tools
• Chef
• Puppet
• Ansible
• Salt
• CFEngine
✨ @beerops(@sigje(#effec/vedevops✨ 69
Introduc)on*to*Chef
✨ @beerops(@sigje(#effec/vedevops✨ 70
Resources
• Ingredients+of+infrastructure
• Basic+building+blocks
✨ @beerops(@sigje(#effec/vedevops✨ 71
Resource(Declara,on
RESOURCETYPE "RESOURCE_NAME" do
PARAMETER PARAMETER_VALUE
end
✨ @beerops(@sigje(#effec/vedevops✨ 72
Example(Resource(Type(1(package
A"package"to"be"installed:
package "httpd" do
action :install
end
✨ @beerops(@sigje(#effec/...
Example(Resource(Type(1(service
A"service"that"should"be"started:
service "httpd" do
supports :restart => :true
action [:e...
Resources
A"resource"is"a"statement"of"policy"that:
• Describes)the)desired)state)for)an)element
• Specifies)a)resource)typ...
Recipes
• Collec'on)of)ordered)resources
• Combina'on)of)ruby)and)Chef)DSL
✨ @beerops(@sigje(#effec/vedevops✨ 76
Cookbooks
• Thema'c
• Collec'on-of-recipes-and-other-suppor'ng-files
✨ @beerops(@sigje(#effec/vedevops✨ 77
Roles
• Abstrac)on,describing,func)on,of,system
• Name
• Descrip)on
• Run,list,(ordered,list,of,recipes,and,roles)
✨ @beer...
Run$List
• Ordered&list&of&recipes&and&roles
• Specific&to&a&node
✨ @beerops(@sigje(#effec/vedevops✨ 79
Nodes
• Machine)(virtual,)physical,)cloud)server,)or)other)device))that)is)
managed)by)Chef
✨ @beerops(@sigje(#effec/vedevo...
Environments
• Abstrac)on,models,workflow
• Name
• Descrip)on
• Cookbook,version,pinning
✨ @beerops(@sigje(#effec/vedevops✨ ...
Supermarket
• Community*site*with*a*number*of*cookbooks
• Read*before*using*in*your*environment
✨ @beerops(@sigje(#effec/ve...
Chef%DK
• Chef&development&kit
• Includes&a&number&of&u8li8es&and&so9ware&to&facilitate&cookbook&
crea8on
• Free&download&...
Berkshelf
• Dependency)management
• Included)with)Chef)DK
✨ @beerops(@sigje(#effec/vedevops✨ 84
Test%Kitchen
• Included)with)Chef)DK
• Sandbox)automa8on
• Test)harness
✨ @beerops(@sigje(#effec/vedevops✨ 85
Test%Kitchen
• Execute(code(on(one(or(more(pla1orms
• Driver(plugins(suppor7ng(various(cloud(and(virtualiza7on(
providers
...
.kitchen.yml
• driver
• provisioner
• pla-orms
• suites
✨ @beerops(@sigje(#effec/vedevops✨ 87
.kitchen.yml,driver
• virtualiza*on-or-cloud-provider
Example:)vagrant,)docker
✨ @beerops(@sigje(#effec/vedevops✨ 88
.kitchen.yml,provisioner
• applica'on*to*configure*the*node
Example:)chef_zero
✨ @beerops(@sigje(#effec/vedevops✨ 89
.kitchen.yml,pla/orms
• target'opera*ng'systems
Example:)centos/6.5
✨ @beerops(@sigje(#effec/vedevops✨ 90
.kitchen.yml,suites
• target'configura-ons
Example:
name:default
run_list:
- recipe[apache::default]
attributes:
✨ @beerops...
Kitchen(commands((1/2)
• kitchen)init
• kitchen)list
• kitchen)create
• kitchen)converge
✨ @beerops(@sigje(#effec/vedevops✨...
Kitchen(commands((2/2)
• kitchen)verify
• kitchen)destroy
• kitchen)test
✨ @beerops(@sigje(#effec/vedevops✨ 93
Docker
• Images
• Registries
• Containers
✨ @beerops(@sigje(#effec/vedevops✨ 94
Introduc)on*to*Lab*3
Lab$3
✨ @beerops(@sigje(#effec/vedevops✨ 95
Lab$3
Time:&30&minutes
✨ @beerops(@sigje(#effec/vedevops✨ 96
✨ @beerops(@sigje(#effec/vedevops✨ 97
Managing&Risk
• Test
• Small*frequent*releases
✨ @beerops(@sigje(#effec/vedevops✨ 98
Lin$ng
• Ensure(code(adheres(to(styles(and(conven2ons
• Weave(expecta2ons(into(development
• Encourages(collabora2on
✨ @be...
Tes$ng
• Documen)ng+objec)ves+and+intent
• Measuring+"done"
✨ @beerops(@sigje(#effec/vedevops✨ 100
Code%Correctness
• foodcri(c
• rubocop
✨ @beerops(@sigje(#effec/vedevops✨ 101
Integra(on*Tests
• ServerSpec
✨ @beerops(@sigje(#effec/vedevops✨ 102
Rubocop
• Ruby&linter
• Ruby&style&guide
• Included&with&ChefDK
✨ @beerops(@sigje(#effec/vedevops✨ 103
Rubocop'Example
$ rubocop cookbooks/COOKBOOK1 cookbooks/COOKBOOK2 cookbooks/COOKBOOK4
✨ @beerops(@sigje(#effec/vedevops✨ 104
Reading(Rubocop(Output
Inspecting 8 files
CWCWCCCC
• ."means"that"the"file"contains"no"issues
• C"means"a"issue"with"conven...
Disabling)Rubocop)cops
Any$configura,on$in$.rubocop.yml$is$disabled.
To#disable#string#literals:
StringLiterals:
Enabled: f...
Foodcri'c
• Chef&linter
• Chef&style&guide
• Included&with&ChefDK
✨ @beerops(@sigje(#effec/vedevops✨ 107
Foodcri'c(Example
$ foodcritic cookbooks/setup
✨ @beerops(@sigje(#effec/vedevops✨ 108
Reading(Foodcri-c(Output
FC008: Generated cookbook metadata needs updating: ./metadata.rb:2
✨ @beerops(@sigje(#effec/vedevo...
ServerSpec
• Tests&to&verify&servers&func0onality
• Resource&types
• Package,&service,&user,&and&many&others
• Integrates&...
ServerSpec'Generic'Form
describe "<subject>" do
it "<description>" do
expect(thing).to eq result
end
end
✨ @beerops(@sigje...
ServerSpec'Poten,al'Tests
• Is$the$service$running?
• Is$the$port$accessible?
• Is$the$expected$content$being$served?
✨ @b...
ServerSpec'Example
describe 'apache' do
it "is installed" do
expect(package 'httpd').to be_installed
end
it "is running" d...
Reading(ServerSpec(Output
app::default
httpd service is running
Finished in 0.26429 seconds (files took 0.7166 seconds to ...
✨ @beerops(@sigje(#effec/vedevops✨ 115
Introduc)on*to*Lab*4
Lab$4
✨ @beerops(@sigje(#effec/vedevops✨ 116
Lab$4
Time:&30&minutes
✨ @beerops(@sigje(#effec/vedevops✨ 117
Introduc)on*to*Lab*5
Lab$4
✨ @beerops(@sigje(#effec/vedevops✨ 118
Lab$5
Time:&20&minutes
✨ @beerops(@sigje(#effec/vedevops✨ 119
Test,&Monitor,&or&Diagnos/c2
1. Where'is'it'going'to'run?
2. When'is'it'going'to'run?
3. How'o4en'will'it'run?
4. Who'is'g...
Measuring*Impact*and*Value*of*Change
✨ @beerops(@sigje(#effec/vedevops✨ 121
Impact'of'Change
✨ @beerops(@sigje(#effec/vedevops✨ 122
Impact'on'Availability
• Overall(site/app(availability
• Individual(service(availability
✨ @beerops(@sigje(#effec/vedevops✨...
Availability)Monitoring
• Up$me:
• Pingdom,/Moni$s,/Uptrends,/etc
• Ver$cal/Line/Technology:
• Availability/a?er/deploys/c...
✨ @beerops(@sigje(#effec/vedevops✨ 125
Even%nator
✨ @beerops(@sigje(#effec/vedevops✨ 126
✨ @beerops(@sigje(#effec/vedevops✨ 127
Service'Availability
• Nagios:)Service/level)monitoring)and)aler5ng
• Nagios/herald:)Alert)context
• OpsWeekly:)Historical...
Nagios
✨ @beerops(@sigje(#effec/vedevops✨ 129
define command {
command_name check_mongodb_query
command_line $USER1$/nagios-plugin-mongodb/check_mongodb.py
-H $HOSTADDR...
define servicedependency{
host_name WWW1
service_description Apache Web Server
dependent_host_name WWW1
dependent_service_...
Nagios'herald
✨ @beerops(@sigje(#effec/vedevops✨ 132
✨ @beerops(@sigje(#effec/vedevops✨ 133
OpsWeekly
✨ @beerops(@sigje(#effec/vedevops✨ 134
✨ @beerops(@sigje(#effec/vedevops✨ 135
✨ @beerops(@sigje(#effec/vedevops✨ 136
✨ @beerops(@sigje(#effec/vedevops✨ 137
✨ @beerops(@sigje(#effec/vedevops✨ 138
Impact'on'Quality
• Service(quality((SLAs)
• Visibility(of(quality
✨ @beerops(@sigje(#effec/vedevops✨ 139
Statsd
✨ @beerops(@sigje(#effec/vedevops✨ 140
>>> import statsd
>>>
>>> timer = statsd.Timer('MyApplication')
>>>
>>> timer.start()
>>> # do something here
>>> timer.st...
>>> import statsd
>>>
>>> counter = statsd.Counter('MyApplication')
>>> # do something here
>>> counter += 1
✨ @beerops(@s...
>>> import statsd
>>>
>>> average = statsd.Average('MyApplication', connection)
>>> # do something here
>>> average.send('...
Graphite
✨ @beerops(@sigje(#effec/vedevops✨ 144
Value&of&Change
✨ @beerops(@sigje(#effec/vedevops✨ 145
Value&of&Availability
• Be$er&for&customers
• Be$er&for&employees&(internal&services)
• Fewer&pages
✨ @beerops(@sigje(#effe...
Value&of&Quality
• Deploys)take)less)-me
• Also)be1er)for)customers
• More)visibility)into)issues
✨ @beerops(@sigje(#effec/...
Retrospec)ve+!
✨ @beerops(@sigje(#effec/vedevops✨ 148
Review
• Recognizing*your*Devops*Narra4ve
• Applica4on*Deployment*Planning
• Infrastructure*as*code
• Introducing*repeatab...
Next%Steps
• Manual,(Automa-on(to(Con-nuous("X"
• Be(the(storylistener(and(storyteller(in(your(org
• Effec-ve(Devops(availa...
Thank&you!&❤@sigje
@beerops
✨ @beerops(@sigje(#effec/vedevops✨ 151
✨ @beerops(@sigje(#effec/vedevops✨ 152
Upcoming SlideShare
Loading in …5
×

Effective Devops - Velocity New York 2015

1,912 views

Published on

DevOps is a professional and cultural movement designed to improve the processes involved with developing and running computer software and systems. It does this by focusing on communication and collaboration between software developers and operations engineers. DevOps has the potential to improve the products we create as an industry, and the health and well-being of the people creating them.

In this one-day introductory training, you will identify actionable strategies and tools that can be used to leverage DevOps to implement noticeable, long-lasting improvements to your culture and development processes in your environment, regardless of your level and role.

Published in: Technology

Effective Devops - Velocity New York 2015

  1. 1. Effec%ve'Devops: Collabora'on)and)Tools Jennifer'Davis'&'Katherine'Daniels ✨ @beerops(@sigje(#effec/vedevops✨ 1
  2. 2. Jennifer'Davis • So$ware)Engineer,)Chef)So$ware • Co2author)of)"Effec8ve)Devops" • Founder)of)CoffeeOps • DevOpsDays)Silicon)Valley)Organizer ✨ @beerops(@sigje(#effec/vedevops✨ 2
  3. 3. Katherine)Daniels • Senior(Opera,ons(Engineer,(Etsy • Co4author(of("Effec,ve(Devops" • DevopsDays(NYC(Organizer • Ship(Show(Podcast(Co4host ✨ @beerops(@sigje(#effec/vedevops✨ 3
  4. 4. Communica)on • Jennifer(Davis Twi/er:(@sigje Email:(sparklydevops@gmail.com • Katherine(Daniels Twi/er:(@beerops Email:(sparklyyakshaver@gmail.com Twi$er:(#effec,vedevops ✨ @beerops(@sigje(#effec/vedevops✨ 4
  5. 5. Feedback • Construc*ve-feedback • What-did-you-find-helpful? • What-would-you-like-to-see-more/less-of? • Was-there-anything-you-found-unclear? ✨ @beerops(@sigje(#effec/vedevops✨ 5
  6. 6. Schedule • Introduc*on+to+teams,+devops+principles • 10:30911:00am+Morning+Break+9+Su?on+Complex • Visualiza*on+of+work,+Git,+Infrastructure+automa*on • 12:3091:30pm+Lunch+9+Americas+Halls+1+&+2 ✨ @beerops(@sigje(#effec/vedevops✨ 6
  7. 7. Schedule • Tes%ng(infrastucture(automa%on(and(other(changes • 3:0073:30pm(A:ernoon(Break(7(Su>on(Complex • Measuring,(monitoring,(and(wrap7up • Ignites:(Beekman(Parlor((with(Katherine) ✨ @beerops(@sigje(#effec/vedevops✨ 7
  8. 8. Network(Connec,vity Network(Name:(Velocity Access(Code:((none) Local&Downloads:&h.p://10.10.32.101 ✨ @beerops(@sigje(#effec/vedevops✨ 8
  9. 9. Expecta(ons • Safe&space&to&share&experiences,&learn&from&each&other • Code&of&Conduct • Learn&effec9ve&workflows&for&using&and&tes9ng&source&control& and&configura9on&management ✨ @beerops(@sigje(#effec/vedevops✨ 9
  10. 10. Team%Introduc.ons • Meet%your%team! • What%are%mo/va/ons?%What%are%current%beliefs?%What%are% current%skills?%Gaps%in%skills? Time:&20&minutes ✨ @beerops(@sigje(#effec/vedevops✨ 10
  11. 11. What%is%Devops ✨ @beerops(@sigje(#effec/vedevops✨ 11
  12. 12. What%is%Devops Cultural'movement'that'seeks'to: • change(how(individuals(work, • value(the(diversity(of(work(done, • develop(conscious(decisions(in(accelera8on, • plan(for(scale,(and • measure(the(effect(of(social(and(technical(change. ✨ @beerops(@sigje(#effec/vedevops✨ 12
  13. 13. Folk%Models • general(popularly(understood(meaning(par2cular(to(a(socio4 cultural(grouping(but(which(has(not(been(formally(defined(or( standardized. ✨ @beerops(@sigje(#effec/vedevops✨ 13
  14. 14. Why$Devops? ✨ @beerops(@sigje(#effec/vedevops✨ 14
  15. 15. High%Performing%Devops%Teams are$more$agile 30X$more$frequent$deployments 8000X$faster$lead$-mes$than$peers 2014%PuppetLabs%State%of%DevOps%Survey ✨ @beerops(@sigje(#effec/vedevops✨ 15
  16. 16. High%Performing%Devops%Teams are$more$reliable 2X#change#success#rate 12X$faster$mean$-me$to$recovery$(MTTR) 2014%PuppetLabs%State%of%DevOps%Survey ✨ @beerops(@sigje(#effec/vedevops✨ 16
  17. 17. Five%Pillars%of%Effec0ve%Devops • Collabora(on • Hiring • Affinity • Tools • Scaling ✨ @beerops(@sigje(#effec/vedevops✨ 17
  18. 18. Collabora'on • Individuals+Working+Together ✨ @beerops(@sigje(#effec/vedevops✨ 18
  19. 19. Hiring • Choosing)Individuals ✨ @beerops(@sigje(#effec/vedevops✨ 19
  20. 20. Affinity • From&Individuals&to&Teams ✨ @beerops(@sigje(#effec/vedevops✨ 20
  21. 21. Tools • Accelerators+of+Culture ✨ @beerops(@sigje(#effec/vedevops✨ 21
  22. 22. Scaling • Evolu'on,*growth*and*advancement*of*the*organiza'on ✨ @beerops(@sigje(#effec/vedevops✨ 22
  23. 23. Collabora'on)and)Tools ✨ @beerops(@sigje(#effec/vedevops✨ 23
  24. 24. Recognizing)your)Devops)Narra3ve ✨ @beerops(@sigje(#effec/vedevops✨ 24
  25. 25. The$Devops$Compact • shared(mutual(understanding • established(boundaries ✨ @beerops(@sigje(#effec/vedevops✨ 25
  26. 26. Team • Common&purpose • Defined&beliefs • Empowered ✨ @beerops(@sigje(#effec/vedevops✨ 26
  27. 27. Careless'Conversa+ons'(inspired'by'Alan'Cyment) • Pair&up • Select&one&person&to&go&first. • For&one&minute,&speaker&talks&about&something&passionate&about.& Listener&stays&seated,&quiet,&and&acts&disinterested. • Switch&roles&and&repeat. • Repeat&un@l&both&people&have&done&this&twice. Time:&5&minutes ✨ @beerops(@sigje(#effec/vedevops✨ 27
  28. 28. Careless'Conversa+ons • How%does%it%feel%not%to%be%listened%to? • How%does%it%feel%to%ignore%someone? ✨ @beerops(@sigje(#effec/vedevops✨ 28
  29. 29. Cul$va$ng)Empathy • Collect(stories • Listen • Circle(back ✨ @beerops(@sigje(#effec/vedevops✨ 29
  30. 30. Smarter'Teams'Build'Be/er'Value1 • Lots&of&Communica/on • Contribute&equally&to&team's&discussions • Theory&of&Mind • Increased&diversity 1 "Engel,"David"et"al."'Reading"The"Mind"In"The"Eyes"Or"Reading"Between"The"Lines?"Theory"Of"Mind"Predicts" CollecCve"Intelligence"Equally"Well"Online"And"FaceIToIFace'."PLoS"ONE"9.12"(2014):"e115212."Web."26"May"2015. ✨ @beerops(@sigje(#effec/vedevops✨ 30
  31. 31. A"life"becomes"meaningful"when"one"sees"himself"or"herself"as"an" actor"within"the"context"of"a"story. !!"George"Howard ✨ @beerops(@sigje(#effec/vedevops✨ 31
  32. 32. Small%vs%Large%teams • Large'teams'+'roles'may'be'highly'segregated • Small'teams'+'one'person'may'be'responsible'for'many'roles ✨ @beerops(@sigje(#effec/vedevops✨ 32
  33. 33. Cri$cal(Habits(for(Teams • Code&Review • Pairing ✨ @beerops(@sigje(#effec/vedevops✨ 33
  34. 34. Code%Review • Max%90%minutes%in%one%se0ng ✨ @beerops(@sigje(#effec/vedevops✨ 34
  35. 35. Pairing • Agile'so*ware'development • 2'people'work'together'on'1'worksta8on • Driver':'writes'code • Observer':'reviews'each'line • Roles'switch'frequently ✨ @beerops(@sigje(#effec/vedevops✨ 35
  36. 36. Types&of&Pairing • Expert(expert • Expert(novice • Novice(novice ✨ @beerops(@sigje(#effec/vedevops✨ 36
  37. 37. Visualiza(on+of+Work • Bug/issue)queue • Kanban ✨ @beerops(@sigje(#effec/vedevops✨ 37
  38. 38. Factors(for(Success • Management)buy)in • Training • Minimize)'pushing' • Team)sync ✨ @beerops(@sigje(#effec/vedevops✨ 38
  39. 39. Workflow Work%that%is • orchestrated • repeatable • organized • moves4from4one4state4to4another ✨ @beerops(@sigje(#effec/vedevops✨ 39
  40. 40. WIP • Work&in&Progress • work&that&has&had&money&or&people&applied&to&it. ✨ @beerops(@sigje(#effec/vedevops✨ 40
  41. 41. Work%Iden*fica*on • name • start)date • end)date • current)state • descrip/on • priority ✨ @beerops(@sigje(#effec/vedevops✨ 41
  42. 42. Task%Handling • What&is&it? • Can&you&do&anything&with&it? • What&is&the&next&step? ✨ @beerops(@sigje(#effec/vedevops✨ 42
  43. 43. Projects • Same&requirements&as&a&task • Larger&in&scope • May&be&comprised&of&more&than&one&task ✨ @beerops(@sigje(#effec/vedevops✨ 43
  44. 44. Interrupts • Non%planned%work%that%comes%in • customer%request • incident • request%for%help%from%coworker • single%point%of%knowledge%(you)%work • high%priority%task%push%from%manager ✨ @beerops(@sigje(#effec/vedevops✨ 44
  45. 45. Blocked(work • Work&that&can&progress&no&further: • dependent&teams&5&blocked&by&external&team • insufficiently&qualified&request&5&blocked&by&requester • dependent&on&SPOK&5&blocked&by&team • Bme&dependent ✨ @beerops(@sigje(#effec/vedevops✨ 45
  46. 46. Team%Exercise Discuss&with&your&team: • What&is&the&difference&between&a&task&and&a&project? • Do&you&have&interrupts?&What&are&they? • How&do&you&determine&when&work&is&done? Time:&15&minutes ✨ @beerops(@sigje(#effec/vedevops✨ 46
  47. 47. Kanban • Start&with&what&you&do&now • Agree&to&incremental,&evolu7onary&change • Respect • Everyone&is&a&leader ✨ @beerops(@sigje(#effec/vedevops✨ 47
  48. 48. Kanban%Prac)ces • Visualize • Limit-WIP • Manage-flow • Make-policies-explicit • Implement-feedback-loops ✨ @beerops(@sigje(#effec/vedevops✨ 48
  49. 49. Visualize • Intent • Alignment • Coherence ✨ @beerops(@sigje(#effec/vedevops✨ 49
  50. 50. Limit%WIP • Pull%(don't%push) ✨ @beerops(@sigje(#effec/vedevops✨ 50
  51. 51. Manage&Flow • Monitor/measure/report • Incremental2change ✨ @beerops(@sigje(#effec/vedevops✨ 51
  52. 52. Make%Policies%Explicit • Document*processes • Group*signoff ✨ @beerops(@sigje(#effec/vedevops✨ 52
  53. 53. Implement(Feedback(Loops • Collabora(on • Retrospec(ves ✨ @beerops(@sigje(#effec/vedevops✨ 53
  54. 54. Devops'Tools • Establish*local*development*environment • Version*control • Manual*7>*Automa:on*7>*Con:nuous • Ar:facts • Infrastructure • Sandbox ✨ @beerops(@sigje(#effec/vedevops✨ 54
  55. 55. Local&Development&Environment&(LDE) • Consistent)set)of)tools)across)the)team • Ability)to)quickly)onboard)new)engineers ✨ @beerops(@sigje(#effec/vedevops✨ 55
  56. 56. Provisioned*Node*,*LDE • AWS%instance%node • Chef%DK • Test%Kitchen • Ruby • ChefSpec,%ServerSpec • Git ✨ @beerops(@sigje(#effec/vedevops✨ 56
  57. 57. Configura)on*Management • Process(of(iden-fying,(managing,(monitoring,(and(audi-ng(a( product(through(its(en-re(life(including(the(processes,( documenta-on,(people,(tools,(so8ware,(and(systems. ✨ @beerops(@sigje(#effec/vedevops✨ 57
  58. 58. Version(Control • Records)changes)to)files)or)sets)of)files)stored)within)the)system • Enable)revisions • Integrity)checking • Collabora<on ✨ @beerops(@sigje(#effec/vedevops✨ 58
  59. 59. Ar#fact(Repository • Secure • Trusted • Stable • Accessible • Versioned ✨ @beerops(@sigje(#effec/vedevops✨ 59
  60. 60. Introduc)on*to*Lab*1 Lab$1 User:&chef Password:&chef ✨ @beerops(@sigje(#effec/vedevops✨ 60
  61. 61. Lab$1 Lab$1 Time:&15&minutes ✨ @beerops(@sigje(#effec/vedevops✨ 61
  62. 62. Introduc)on*to*Lab*2 Lab$2 ✨ @beerops(@sigje(#effec/vedevops✨ 62
  63. 63. Lab$2 Lab$2 Time:&15&minutes ✨ @beerops(@sigje(#effec/vedevops✨ 63
  64. 64. Welcome'Back • git%review% ✨ @beerops(@sigje(#effec/vedevops✨ 64
  65. 65. git$pull git$pull$REMOTE ✨ @beerops(@sigje(#effec/vedevops✨ 65
  66. 66. git$push$remote$branch • transfer)commits)from)a)local)repo)to)a)remote)repo. • counterpart)to)git)fetch ✨ @beerops(@sigje(#effec/vedevops✨ 66
  67. 67. Infrastructure • Aggregate(of(applica/ons,(configura/ons,(access(control,(data,( compute(nodes,(network,(storage,(processes,(and(people. ✨ @beerops(@sigje(#effec/vedevops✨ 67
  68. 68. Infrastructure+Automa/on • Systems(that(reduce(the(burden(on(people(to(manage(services( and(increase(the(quality,(accuracy(and(precision(of(a(service(to( the(consumers(of(a(service ✨ @beerops(@sigje(#effec/vedevops✨ 68
  69. 69. Infrastructure+Automa/on+Tools • Chef • Puppet • Ansible • Salt • CFEngine ✨ @beerops(@sigje(#effec/vedevops✨ 69
  70. 70. Introduc)on*to*Chef ✨ @beerops(@sigje(#effec/vedevops✨ 70
  71. 71. Resources • Ingredients+of+infrastructure • Basic+building+blocks ✨ @beerops(@sigje(#effec/vedevops✨ 71
  72. 72. Resource(Declara,on RESOURCETYPE "RESOURCE_NAME" do PARAMETER PARAMETER_VALUE end ✨ @beerops(@sigje(#effec/vedevops✨ 72
  73. 73. Example(Resource(Type(1(package A"package"to"be"installed: package "httpd" do action :install end ✨ @beerops(@sigje(#effec/vedevops✨ 73
  74. 74. Example(Resource(Type(1(service A"service"that"should"be"started: service "httpd" do supports :restart => :true action [:enable, :start] end ✨ @beerops(@sigje(#effec/vedevops✨ 74
  75. 75. Resources A"resource"is"a"statement"of"policy"that: • Describes)the)desired)state)for)an)element • Specifies)a)resource)type888such)as)package,)template,)or) service • Lists)addi;onal)details)(also)known)as)parameters),)as)necessary • Are)grouped)into)recipes ✨ @beerops(@sigje(#effec/vedevops✨ 75
  76. 76. Recipes • Collec'on)of)ordered)resources • Combina'on)of)ruby)and)Chef)DSL ✨ @beerops(@sigje(#effec/vedevops✨ 76
  77. 77. Cookbooks • Thema'c • Collec'on-of-recipes-and-other-suppor'ng-files ✨ @beerops(@sigje(#effec/vedevops✨ 77
  78. 78. Roles • Abstrac)on,describing,func)on,of,system • Name • Descrip)on • Run,list,(ordered,list,of,recipes,and,roles) ✨ @beerops(@sigje(#effec/vedevops✨ 78
  79. 79. Run$List • Ordered&list&of&recipes&and&roles • Specific&to&a&node ✨ @beerops(@sigje(#effec/vedevops✨ 79
  80. 80. Nodes • Machine)(virtual,)physical,)cloud)server,)or)other)device))that)is) managed)by)Chef ✨ @beerops(@sigje(#effec/vedevops✨ 80
  81. 81. Environments • Abstrac)on,models,workflow • Name • Descrip)on • Cookbook,version,pinning ✨ @beerops(@sigje(#effec/vedevops✨ 81
  82. 82. Supermarket • Community*site*with*a*number*of*cookbooks • Read*before*using*in*your*environment ✨ @beerops(@sigje(#effec/vedevops✨ 82
  83. 83. Chef%DK • Chef&development&kit • Includes&a&number&of&u8li8es&and&so9ware&to&facilitate&cookbook& crea8on • Free&download&off&of&the&website ✨ @beerops(@sigje(#effec/vedevops✨ 83
  84. 84. Berkshelf • Dependency)management • Included)with)Chef)DK ✨ @beerops(@sigje(#effec/vedevops✨ 84
  85. 85. Test%Kitchen • Included)with)Chef)DK • Sandbox)automa8on • Test)harness ✨ @beerops(@sigje(#effec/vedevops✨ 85
  86. 86. Test%Kitchen • Execute(code(on(one(or(more(pla1orms • Driver(plugins(suppor7ng(various(cloud(and(virtualiza7on( providers ✨ @beerops(@sigje(#effec/vedevops✨ 86
  87. 87. .kitchen.yml • driver • provisioner • pla-orms • suites ✨ @beerops(@sigje(#effec/vedevops✨ 87
  88. 88. .kitchen.yml,driver • virtualiza*on-or-cloud-provider Example:)vagrant,)docker ✨ @beerops(@sigje(#effec/vedevops✨ 88
  89. 89. .kitchen.yml,provisioner • applica'on*to*configure*the*node Example:)chef_zero ✨ @beerops(@sigje(#effec/vedevops✨ 89
  90. 90. .kitchen.yml,pla/orms • target'opera*ng'systems Example:)centos/6.5 ✨ @beerops(@sigje(#effec/vedevops✨ 90
  91. 91. .kitchen.yml,suites • target'configura-ons Example: name:default run_list: - recipe[apache::default] attributes: ✨ @beerops(@sigje(#effec/vedevops✨ 91
  92. 92. Kitchen(commands((1/2) • kitchen)init • kitchen)list • kitchen)create • kitchen)converge ✨ @beerops(@sigje(#effec/vedevops✨ 92
  93. 93. Kitchen(commands((2/2) • kitchen)verify • kitchen)destroy • kitchen)test ✨ @beerops(@sigje(#effec/vedevops✨ 93
  94. 94. Docker • Images • Registries • Containers ✨ @beerops(@sigje(#effec/vedevops✨ 94
  95. 95. Introduc)on*to*Lab*3 Lab$3 ✨ @beerops(@sigje(#effec/vedevops✨ 95
  96. 96. Lab$3 Time:&30&minutes ✨ @beerops(@sigje(#effec/vedevops✨ 96
  97. 97. ✨ @beerops(@sigje(#effec/vedevops✨ 97
  98. 98. Managing&Risk • Test • Small*frequent*releases ✨ @beerops(@sigje(#effec/vedevops✨ 98
  99. 99. Lin$ng • Ensure(code(adheres(to(styles(and(conven2ons • Weave(expecta2ons(into(development • Encourages(collabora2on ✨ @beerops(@sigje(#effec/vedevops✨ 99
  100. 100. Tes$ng • Documen)ng+objec)ves+and+intent • Measuring+"done" ✨ @beerops(@sigje(#effec/vedevops✨ 100
  101. 101. Code%Correctness • foodcri(c • rubocop ✨ @beerops(@sigje(#effec/vedevops✨ 101
  102. 102. Integra(on*Tests • ServerSpec ✨ @beerops(@sigje(#effec/vedevops✨ 102
  103. 103. Rubocop • Ruby&linter • Ruby&style&guide • Included&with&ChefDK ✨ @beerops(@sigje(#effec/vedevops✨ 103
  104. 104. Rubocop'Example $ rubocop cookbooks/COOKBOOK1 cookbooks/COOKBOOK2 cookbooks/COOKBOOK4 ✨ @beerops(@sigje(#effec/vedevops✨ 104
  105. 105. Reading(Rubocop(Output Inspecting 8 files CWCWCCCC • ."means"that"the"file"contains"no"issues • C"means"a"issue"with"conven2on • W"means"a"warning • E"means"an"error • F"means"an"fatal"error ✨ @beerops(@sigje(#effec/vedevops✨ 105
  106. 106. Disabling)Rubocop)cops Any$configura,on$in$.rubocop.yml$is$disabled. To#disable#string#literals: StringLiterals: Enabled: false ✨ @beerops(@sigje(#effec/vedevops✨ 106
  107. 107. Foodcri'c • Chef&linter • Chef&style&guide • Included&with&ChefDK ✨ @beerops(@sigje(#effec/vedevops✨ 107
  108. 108. Foodcri'c(Example $ foodcritic cookbooks/setup ✨ @beerops(@sigje(#effec/vedevops✨ 108
  109. 109. Reading(Foodcri-c(Output FC008: Generated cookbook metadata needs updating: ./metadata.rb:2 ✨ @beerops(@sigje(#effec/vedevops✨ 109
  110. 110. ServerSpec • Tests&to&verify&servers&func0onality • Resource&types • Package,&service,&user,&and&many&others • Integrates&with&Test&Kitchen • h?p://serverspec.org ✨ @beerops(@sigje(#effec/vedevops✨ 110
  111. 111. ServerSpec'Generic'Form describe "<subject>" do it "<description>" do expect(thing).to eq result end end ✨ @beerops(@sigje(#effec/vedevops✨ 111
  112. 112. ServerSpec'Poten,al'Tests • Is$the$service$running? • Is$the$port$accessible? • Is$the$expected$content$being$served? ✨ @beerops(@sigje(#effec/vedevops✨ 112
  113. 113. ServerSpec'Example describe 'apache' do it "is installed" do expect(package 'httpd').to be_installed end it "is running" do expect(service 'httpd').to be_running end end ✨ @beerops(@sigje(#effec/vedevops✨ 113
  114. 114. Reading(ServerSpec(Output app::default httpd service is running Finished in 0.26429 seconds (files took 0.7166 seconds to load) 1 example, 0 failures ✨ @beerops(@sigje(#effec/vedevops✨ 114
  115. 115. ✨ @beerops(@sigje(#effec/vedevops✨ 115
  116. 116. Introduc)on*to*Lab*4 Lab$4 ✨ @beerops(@sigje(#effec/vedevops✨ 116
  117. 117. Lab$4 Time:&30&minutes ✨ @beerops(@sigje(#effec/vedevops✨ 117
  118. 118. Introduc)on*to*Lab*5 Lab$4 ✨ @beerops(@sigje(#effec/vedevops✨ 118
  119. 119. Lab$5 Time:&20&minutes ✨ @beerops(@sigje(#effec/vedevops✨ 119
  120. 120. Test,&Monitor,&or&Diagnos/c2 1. Where'is'it'going'to'run? 2. When'is'it'going'to'run? 3. How'o4en'will'it'run? 4. Who'is'going'to'consume'the'result? 5. What'is'the'en;ty'going'to'do'with'it? 2 "Lam,"Yvonne."'Sysadvent:"Day"5"6"How"To"Talk"About"Monitors,"Tests,"And"DiagnosCcs'."Sysadvent.blogspot.com." N.p.,"2014."Web."26"May"2015. ✨ @beerops(@sigje(#effec/vedevops✨ 120
  121. 121. Measuring*Impact*and*Value*of*Change ✨ @beerops(@sigje(#effec/vedevops✨ 121
  122. 122. Impact'of'Change ✨ @beerops(@sigje(#effec/vedevops✨ 122
  123. 123. Impact'on'Availability • Overall(site/app(availability • Individual(service(availability ✨ @beerops(@sigje(#effec/vedevops✨ 123
  124. 124. Availability)Monitoring • Up$me: • Pingdom,/Moni$s,/Uptrends,/etc • Ver$cal/Line/Technology: • Availability/a?er/deploys/changes ✨ @beerops(@sigje(#effec/vedevops✨ 124
  125. 125. ✨ @beerops(@sigje(#effec/vedevops✨ 125
  126. 126. Even%nator ✨ @beerops(@sigje(#effec/vedevops✨ 126
  127. 127. ✨ @beerops(@sigje(#effec/vedevops✨ 127
  128. 128. Service'Availability • Nagios:)Service/level)monitoring)and)aler5ng • Nagios/herald:)Alert)context • OpsWeekly:)Historical)alert)data ✨ @beerops(@sigje(#effec/vedevops✨ 128
  129. 129. Nagios ✨ @beerops(@sigje(#effec/vedevops✨ 129
  130. 130. define command { command_name check_mongodb_query command_line $USER1$/nagios-plugin-mongodb/check_mongodb.py -H $HOSTADDRESS$ -A $ARG1$ -P $ARG2$ -W $ARG3$ -C $ARG4$ -q $ARG5$ } define service { use generic-service hostgroup_name Mongo Servers service_description Mongo Connect Check check_command check_mongodb!connect!27017!2!4 } ✨ @beerops(@sigje(#effec/vedevops✨ 130
  131. 131. define servicedependency{ host_name WWW1 service_description Apache Web Server dependent_host_name WWW1 dependent_service_description Main Web Site execution_failure_criteria n notification_failure_criteria w,u,c } ✨ @beerops(@sigje(#effec/vedevops✨ 131
  132. 132. Nagios'herald ✨ @beerops(@sigje(#effec/vedevops✨ 132
  133. 133. ✨ @beerops(@sigje(#effec/vedevops✨ 133
  134. 134. OpsWeekly ✨ @beerops(@sigje(#effec/vedevops✨ 134
  135. 135. ✨ @beerops(@sigje(#effec/vedevops✨ 135
  136. 136. ✨ @beerops(@sigje(#effec/vedevops✨ 136
  137. 137. ✨ @beerops(@sigje(#effec/vedevops✨ 137
  138. 138. ✨ @beerops(@sigje(#effec/vedevops✨ 138
  139. 139. Impact'on'Quality • Service(quality((SLAs) • Visibility(of(quality ✨ @beerops(@sigje(#effec/vedevops✨ 139
  140. 140. Statsd ✨ @beerops(@sigje(#effec/vedevops✨ 140
  141. 141. >>> import statsd >>> >>> timer = statsd.Timer('MyApplication') >>> >>> timer.start() >>> # do something here >>> timer.stop('SomeTimer') ✨ @beerops(@sigje(#effec/vedevops✨ 141
  142. 142. >>> import statsd >>> >>> counter = statsd.Counter('MyApplication') >>> # do something here >>> counter += 1 ✨ @beerops(@sigje(#effec/vedevops✨ 142
  143. 143. >>> import statsd >>> >>> average = statsd.Average('MyApplication', connection) >>> # do something here >>> average.send('SomeName', 'somekey:%d'.format(value)) ✨ @beerops(@sigje(#effec/vedevops✨ 143
  144. 144. Graphite ✨ @beerops(@sigje(#effec/vedevops✨ 144
  145. 145. Value&of&Change ✨ @beerops(@sigje(#effec/vedevops✨ 145
  146. 146. Value&of&Availability • Be$er&for&customers • Be$er&for&employees&(internal&services) • Fewer&pages ✨ @beerops(@sigje(#effec/vedevops✨ 146
  147. 147. Value&of&Quality • Deploys)take)less)-me • Also)be1er)for)customers • More)visibility)into)issues ✨ @beerops(@sigje(#effec/vedevops✨ 147
  148. 148. Retrospec)ve+! ✨ @beerops(@sigje(#effec/vedevops✨ 148
  149. 149. Review • Recognizing*your*Devops*Narra4ve • Applica4on*Deployment*Planning • Infrastructure*as*code • Introducing*repeatable,*testable*change • Measuring*impact*and*value*of*change ✨ @beerops(@sigje(#effec/vedevops✨ 149
  150. 150. Next%Steps • Manual,(Automa-on(to(Con-nuous("X" • Be(the(storylistener(and(storyteller(in(your(org • Effec-ve(Devops(available(in(Early(Release ✨ @beerops(@sigje(#effec/vedevops✨ 150
  151. 151. Thank&you!&❤@sigje @beerops ✨ @beerops(@sigje(#effec/vedevops✨ 151
  152. 152. ✨ @beerops(@sigje(#effec/vedevops✨ 152

×