Join the conversation #devseccon
By Matt Carroll
“Security patching
system packages is
fun!” said no-one ever.
Matt Carroll - Site Reliability Engineer
mattc@yelp.com
@oholiab
“Security patching system packages
is fun!”
said no-one ever.
Some security tasks are just a drag
Social engineering is for protagonists
too!
Use tooling to minimise pain and
maximise motivation
You CAN use technology to help solve
people problems
Lol package management.
The Takeaway
WHO?
mattc
Qualifications:
●Worries too much
Who’s this then?
https://www.flickr.com/photos/ajc1/10994593713
Yelp’s Mission
Connecting people with great
local businesses.
Yelp Stats
As of Q2 2016
92M 3272%108M
Building our PaaS: PaaSTA!
Managing our edge
Supporting deploys and
developer workflows
Server/instance maintenance
Tooling
The kitchen drawer
Backronyming badly
Not just rebooting and saying no.
Operations team
KC Green
heartbleed.com
KC Green
Patching packages is hard and boring
Things that are difficult to upgrade in place without
downtime
Technical debt and edge cases
Package freezing
Yelppacks
FrankenLucid
Docker
Trying to do clever things with apt
So what’s the problem?
Doing clever things
with apt
http://scarfolk.blogspot.co.uk/2013/05/the-dont-campaign-and-kak-1973.html
#!/bin/bash
[ -z "$1" ] && exit 2
library_regex="$@"
mapped_deleted=""
while read process; do
pid=$(echo $process | awk '{print $1}')
cmd=$(echo $process | awk '{print $2}')
map=$(sudo grep -E "$library_regex" /proc/$pid/maps 2>/dev/null)
if ! [ "$map" = "" ]; then
echo -e "n${process}n------------"
echo "$map"
if echo "$map" | grep -q "(deleted)"; then
mapped_deleted="$mapped_deletedn$cmd"
fi
fi
done< <(ps --no-header -eo pid,comm)
if [ "$mapped_deleted" = "" ]; then
exit 1
else
echo
echo "NEEDS RESTART"
echo -n "============="
echo -e "$mapped_deleted" | sort | uniq
exit 0
fi
IN B4 APT-GET
UPGRADE
For tech debt
For reproducible builds
Because apt does silly things
To generally be aware of impact of CVEs on critical
components
At least until we’ve built confidence
But it’s really really dull!
Needs eyes on from an engineer
A JIRA project
Ubuntu Security Notice emails straight to tickets
Wheel of Misfortune
Enter AUTOSEC
Engineers tend to like:
Interesting work
New things
To know where they stand
A tight feedback loop
Agency
Tedious stuff is tedious
Engineers tend to like:
Interesting work
New things
Feedback
Agency
Tedious stuff is tedious
https://pixabay.com/p-155981
Engineers tend to like:
Interesting work ❌
New things ❌
Feedback ❌
Agency ❌
Tedious stuff is tedious
Engineers tend to like:
Interesting work ❌
New things ❌
Feedback ❌
Agency ❌
Tedious stuff is tedious
I am not a wizard:
Interesting work ❌
New things ❌
Feedback ✅
Agency ✅
What do I think we can fix?
Aim to reduce MTTR for
security tickets within Q2
Resolution within 2 weeks
unless critical
Close out all pre-Q2 2016
tickets
We had organizational buy in
Already we have a better idea
of where we stand as a team.
Enter AUTOSEC OKR
Automate distribution of work
Including deadlines
Tighten up feedback loop with metrics and frequent
reports
Make the critical path to decisive action more explicit
Improve documentation
Make it easy to get help
Improve perceived agency
How?
Asking non-security specialists to make security decisions
“Won’t Fix” is against engineering nature
Prevent naive interventionism
You are making tradeoffs (absolute security vs moving
faster than competitors)
You only find out if you did the wrong thing
Empower people to make hard decisions with little payoff
autosec-review mail group (leveraging JIRA again)
Anything you can do to make it less painful
Increase Agency: Recognise the futility
JIRA gives us a bunch of
stuff for free
We totally have a PaaS to
put the Wheel of
Misfortune on!
(You could totally do
this with a cron job)
The AUTOSEC service
Also hooks into JIRA
Work distribution
github.com/Netflix-Skunkworks/go-jira
for ad-hoc metrics and mailouts
Helps team members know that they’re
helping and what progress on the goals
are like
Pretty much the only feedback you get
Feedback: Metrics and reporting
Proactively security patching system packages often feels more like an arcane
ritual to satisfy the script kiddie gods than it does engineering. In part, this is
because of a feedback loop that’s more of a feedback line… Post completion,
you’re safe in the knowledge that you still haven’t been hacked that you’re aw
of. Probably.
But it’s still important – if your OS vendor has gotten round to announcing an
fixing vulnerabilities to you, then they’ve landed in everyone else’s inbox too!
This talk will address some of the problems inherent in defensive infrastructu
security. It will give examples of how to change the problem space in order to
motivate engineers toward being proactive in a field that is “everyone’s
responsibility”. Hopefully this should give some insight into how you can
leverage technology and pragmatism to instigate change in your security cultu
By reducing the pain and uncertainty of taking action, you can make
Clear up documentation on process
Even so, a well defined process on paper is difficult to
follow
Did it anyway
Increase agency by REMOVING extraneous information
Break points should happen as early as possible
Should ideally be scripted
Increase Agency: Critical path
Clear up documentation on process
Even so, a well defined process on paper is difficult to
follow
Did it anyway
Increase agency by REMOVING extraneous information
Break points should happen as early as possible
Should ideally be scripted
Increase Agency: Critical path
Deadlines really help you prioritize
work
JIRA and cron(ish) again
Extension of AUTOSEC service
Tells you when you’re nearing
deadline
“I need it done now/ASAP/yesterday”
are not deadlines
Helps to balance against the actually
rewarding work
Increase Agency: nagbot
DID IT WORK?
WHAT NEXT?
Those stats are probably lies
I hope none of our servers run on Snapdragon kernels…
Scrape information out of USNs
Auto-triage information
Introspect with mcollective and what’s in our repos
Maybe even auto-close?
Feed information to documentation scripts to remove
some of the questions
Pointless overhead
When processes are in flux, docs
change
I don’t even want to read the docs
once, and neither do you
Continue on with scripting work
No extraneous information
Process is more interactive
Process changes can be
reviewed for greater
confidence!
Mental caching
http://www.express.co.uk/finance/cr
usader/623732/Crusader-act-now-
victim-PPI
Remove our old cruft (obviously)
Make puppet dpkg pin versions
of packages we install via
puppet
apt-get upgrade from upstream
security becomes safer
Less complicated process means
less can fall through the gaps
Fix packaging
Some security tasks are just a drag
Social engineering is for protagonists
too!
Use tooling to minimise pain and
maximise motivation
You CAN use technology to help solve
people problems
Lol package management.
The Takeaway
Thanks for listening!
@YelpEngineering
fb.com/YelpEngineers
engineeringblog.yelp.com
github.com/yelp
Join the conversation #devseccon
Matt Carroll
SRE at Yelp
mattc@yelp.com
@oholiab
oholiab on Freenode

Matt carroll - "Security patching system packages is fun" said no-one ever

  • 1.
    Join the conversation#devseccon By Matt Carroll “Security patching system packages is fun!” said no-one ever.
  • 2.
    Matt Carroll -Site Reliability Engineer mattc@yelp.com @oholiab “Security patching system packages is fun!” said no-one ever.
  • 3.
    Some security tasksare just a drag Social engineering is for protagonists too! Use tooling to minimise pain and maximise motivation You CAN use technology to help solve people problems Lol package management. The Takeaway
  • 4.
  • 5.
    mattc Qualifications: ●Worries too much Who’sthis then? https://www.flickr.com/photos/ajc1/10994593713
  • 6.
    Yelp’s Mission Connecting peoplewith great local businesses.
  • 7.
    Yelp Stats As ofQ2 2016 92M 3272%108M
  • 8.
    Building our PaaS:PaaSTA! Managing our edge Supporting deploys and developer workflows Server/instance maintenance Tooling The kitchen drawer Backronyming badly Not just rebooting and saying no. Operations team
  • 9.
  • 10.
  • 11.
  • 12.
    Patching packages ishard and boring Things that are difficult to upgrade in place without downtime Technical debt and edge cases Package freezing Yelppacks FrankenLucid Docker Trying to do clever things with apt So what’s the problem?
  • 13.
    Doing clever things withapt http://scarfolk.blogspot.co.uk/2013/05/the-dont-campaign-and-kak-1973.html
  • 14.
    #!/bin/bash [ -z "$1"] && exit 2 library_regex="$@" mapped_deleted="" while read process; do pid=$(echo $process | awk '{print $1}') cmd=$(echo $process | awk '{print $2}') map=$(sudo grep -E "$library_regex" /proc/$pid/maps 2>/dev/null) if ! [ "$map" = "" ]; then echo -e "n${process}n------------" echo "$map" if echo "$map" | grep -q "(deleted)"; then mapped_deleted="$mapped_deletedn$cmd" fi fi done< <(ps --no-header -eo pid,comm) if [ "$mapped_deleted" = "" ]; then exit 1 else echo echo "NEEDS RESTART" echo -n "=============" echo -e "$mapped_deleted" | sort | uniq exit 0 fi IN B4 APT-GET UPGRADE
  • 15.
    For tech debt Forreproducible builds Because apt does silly things To generally be aware of impact of CVEs on critical components At least until we’ve built confidence But it’s really really dull! Needs eyes on from an engineer
  • 16.
    A JIRA project UbuntuSecurity Notice emails straight to tickets Wheel of Misfortune Enter AUTOSEC
  • 17.
    Engineers tend tolike: Interesting work New things To know where they stand A tight feedback loop Agency Tedious stuff is tedious
  • 18.
    Engineers tend tolike: Interesting work New things Feedback Agency Tedious stuff is tedious https://pixabay.com/p-155981
  • 19.
    Engineers tend tolike: Interesting work ❌ New things ❌ Feedback ❌ Agency ❌ Tedious stuff is tedious
  • 20.
    Engineers tend tolike: Interesting work ❌ New things ❌ Feedback ❌ Agency ❌ Tedious stuff is tedious
  • 21.
    I am nota wizard: Interesting work ❌ New things ❌ Feedback ✅ Agency ✅ What do I think we can fix?
  • 22.
    Aim to reduceMTTR for security tickets within Q2 Resolution within 2 weeks unless critical Close out all pre-Q2 2016 tickets We had organizational buy in Already we have a better idea of where we stand as a team. Enter AUTOSEC OKR
  • 23.
    Automate distribution ofwork Including deadlines Tighten up feedback loop with metrics and frequent reports Make the critical path to decisive action more explicit Improve documentation Make it easy to get help Improve perceived agency How?
  • 24.
    Asking non-security specialiststo make security decisions “Won’t Fix” is against engineering nature Prevent naive interventionism You are making tradeoffs (absolute security vs moving faster than competitors) You only find out if you did the wrong thing Empower people to make hard decisions with little payoff autosec-review mail group (leveraging JIRA again) Anything you can do to make it less painful Increase Agency: Recognise the futility
  • 25.
    JIRA gives usa bunch of stuff for free We totally have a PaaS to put the Wheel of Misfortune on! (You could totally do this with a cron job) The AUTOSEC service Also hooks into JIRA Work distribution
  • 26.
    github.com/Netflix-Skunkworks/go-jira for ad-hoc metricsand mailouts Helps team members know that they’re helping and what progress on the goals are like Pretty much the only feedback you get Feedback: Metrics and reporting
  • 27.
    Proactively security patchingsystem packages often feels more like an arcane ritual to satisfy the script kiddie gods than it does engineering. In part, this is because of a feedback loop that’s more of a feedback line… Post completion, you’re safe in the knowledge that you still haven’t been hacked that you’re aw of. Probably. But it’s still important – if your OS vendor has gotten round to announcing an fixing vulnerabilities to you, then they’ve landed in everyone else’s inbox too! This talk will address some of the problems inherent in defensive infrastructu security. It will give examples of how to change the problem space in order to motivate engineers toward being proactive in a field that is “everyone’s responsibility”. Hopefully this should give some insight into how you can leverage technology and pragmatism to instigate change in your security cultu By reducing the pain and uncertainty of taking action, you can make Clear up documentation on process Even so, a well defined process on paper is difficult to follow Did it anyway Increase agency by REMOVING extraneous information Break points should happen as early as possible Should ideally be scripted Increase Agency: Critical path
  • 28.
    Clear up documentationon process Even so, a well defined process on paper is difficult to follow Did it anyway Increase agency by REMOVING extraneous information Break points should happen as early as possible Should ideally be scripted Increase Agency: Critical path
  • 29.
    Deadlines really helpyou prioritize work JIRA and cron(ish) again Extension of AUTOSEC service Tells you when you’re nearing deadline “I need it done now/ASAP/yesterday” are not deadlines Helps to balance against the actually rewarding work Increase Agency: nagbot
  • 30.
  • 33.
  • 34.
    Those stats areprobably lies I hope none of our servers run on Snapdragon kernels… Scrape information out of USNs Auto-triage information Introspect with mcollective and what’s in our repos Maybe even auto-close? Feed information to documentation scripts to remove some of the questions Pointless overhead
  • 35.
    When processes arein flux, docs change I don’t even want to read the docs once, and neither do you Continue on with scripting work No extraneous information Process is more interactive Process changes can be reviewed for greater confidence! Mental caching http://www.express.co.uk/finance/cr usader/623732/Crusader-act-now- victim-PPI
  • 36.
    Remove our oldcruft (obviously) Make puppet dpkg pin versions of packages we install via puppet apt-get upgrade from upstream security becomes safer Less complicated process means less can fall through the gaps Fix packaging
  • 37.
    Some security tasksare just a drag Social engineering is for protagonists too! Use tooling to minimise pain and maximise motivation You CAN use technology to help solve people problems Lol package management. The Takeaway
  • 38.
  • 39.
  • 40.
    Join the conversation#devseccon Matt Carroll SRE at Yelp mattc@yelp.com @oholiab oholiab on Freenode