資安問題牽涉到的層面太過廣範，從線上系統一路到辦公室門禁都被包含在其中 DevOps 的 CAMS 精神其實剛剛好可以跟資安融合在一起，從 Configuration Management, Infrastructure as Code 到最近很紅的 ChatOps 都可以用來幫助加強資安，此分享從大企業與新創一路跌跌撞撞所學以致用的血淚經驗

1. DevOps: Security 干我何事？
1

2. HELLO!
I am smalltown
MaiCoin Site Reliability Engineer
Taipei HashiCorp User Group Organizer
AWS User Group Taiwan Staff
2

3. “
Do You Think Security is
Important in Software
Development?
3

4. “
Everything Goes Well With
Waterfall Model
4

5. “
Agile Model Made Software
Delivery Faster and Faster
5

6. “
It’s Impossible to Test After
Done of Development
6

7. DevOps Power By
Testing
Ref
7

8. “
Security Also Encounter
The Same Problem
8

9. DevOps Power By
Security
Security
Thank for
Coming Today
9

10. Three Elements of
DevSecOps
OperationDeveloper
Information
Security
Security factor into
release pipeline
Unite developers,
security and operation
Increase Visibility
10

11. Outline
⊡ Security Gene in Development
⊡ Secret Management
⊡ Infrastructure As Must Be Code
⊡ System Management
⊡ C.A.S. C.A.M.S.
11

12. Security Gene in
Development
1
12

13. “
Penetration Testing Need
to be Fast and Seamless
with Development Process
13

14. Penetration Test in
Traditional Way
⊡ When new service online, performing
penetration testing By...
□ Information Security Department
□ Third Party Penetration Service
⊡ But … Agile development teams focus on
producing code Not Enhance Security
14

15. Penetration Testing
in Development
⊡ Static application security testing (SAST)
⊡ Runtime application self-protection (RASP)
Ref
15

16. Penetration Testing
in Development
⊡ Dynamic application security testing (DAST)
⊡ Interactive application security testing (IAST)
Ref
16

17. “
A Chain is Only as Strong
as Its Weakest Link
17

18. “
Penetration Test Need
Include Both Human and
System
18

19. Penetration Test for
Human
⊡ Although attend tests/inform orally is not
avoid
⊡ But the exercise make things well imprinted
on human’s brain.
⊡ After all, practice makes perfect
19

20. Penetration Test for
Human
⊡ Don’t Need Strong Knowledge/Skill
⊡ All you need to do is …
□ Leverage Human’s Greedy/Fear
□ Get the organization member contact info
□ Automatic send mail/sms mechanism
(Phish Insight)
□ Host Fake website to collect feedback (e.g.
CredSniper, SET)
20

21. “
Trust Me, The Result Will
Make Your Jaw Drop
21

22. Secret Management
2
22

23. “
Have You Rotated Production
Database Account/Password
Recently?
23

24. Rotate Credentials
⊡ The database credentials will be rotated
when…
□ Database migration
□ External auditing
⊡ Only if the rotate mechanism is implemented
at first, then it truly happens
24

25. HashiCorp Vault
⊡ Secures, stores, and tightly controls access to
tokens, passwords, certificates, API keys
⊡ Handles leasing, key revocation, key rolling,
and auditing
25

26. Ideal Credential
Lifecycle
Service is Accessed
Application
1. Request Access Credential (Running)
2. Use the Credential to Access Service
3. Revoke the Credential
Credentials Only
Exist in Memory
26

27. Authentication
⊡ Vault provide various auth method
□ Tokens, AppRole
□ AWS, Azure, Google Cloud
□ LDAP, GitHub
□ ...etc
27

28. Authorization
⊡ Vault store credentials like key/value
database, e.g.
□ /secret/stag/database/admin
□ /secret/prod/database/admin
⊡ Hence, predefined policy grant appropriate
permission, e.g.
path "secret/stag/database/admin" {
capabilities = ["read"]
}
28

29. Dynamic Credentials
⊡ Vault support many secret backend
□ AWS, Azure, GCP, Database...etc
⊡ Take database for example, you could
generate dynamic database credentials
$ vault read database/creds/my-role
Key Value
--- -----
lease_duration 1h
password 8cab931c-d62e-a73d-60d3-5ee85139cd66
username v-root-e2978cd0-
29

30. Infrastructure As Must Be
Code
3
30

31. “Network Misconfigurations
Are Major Source Of
Reliability and Security
Issues
In a report summarizing the findings of 124 penetration tests,
security firm Rapid7 found that more than two-thirds of sites
were vulnerable because of a misconfiguration
Ref 31

32. “
Why So Many Important
Internal Service Can Be
Accessed Publicly?!
32

33. Keep Server Private
Private Network
- Kubernetes
- Database
- NAT Gateway
- ...
Public Network
- Load Balancer
- Linux Gateway
- ...
- Non-Employee: Only access product service through
load balancer
- Employee: Access server through VPN/Bastion
- Don’t forgot to enable WAF
33

34. “
In Fact, Not Only Network Related
Configuration, Every Program
Misconfiguration Makes Issues
34

35. Infrastructure As
Code
⊡ There are so many benefits when adopting
IaC
□ Save time & Avoid human error
□ Code review & Knowledge transfer easily
□ Testing (kitchen, terratest...etc)
35

36. Infrastructure As
Code
⊡ There are many tools which can achieve IaC
⊡ Servers:
□ Ansible, Chef, Puppet, SaltStack...etc
⊡ Cloud Providers:
□ Terraform, AWS CloudFormation, Azure
Resource Manager, GCP Deployment
Manager...etc
36

37. System Management
4
37

38. 9 Key Point
⊡ Authentication (Later)
⊡ Authorization (Later)
⊡ Secret Management (HashiCorp Vault)
⊡ Don’t Share Account
⊡ Least Privilege Policy
⊡ Log Everything
⊡ Manage and Record Privileged Activity (Later)
⊡ Alert and Notify of Suspicious Activity (Later)
⊡ Identity Centralize and Unify
38

39. Authentication for
Human
⊡ Adopt Password Manager to avoid credential
stuffing attacks
⊡ Must Enable 2FA: What-you-know,
What-you-have and What-you-are (2 out of
the 3 types)
39

40. “
A: I Need Sudo Permission for
Production Deployment Tomorrow
40

41. Manage and Record
Privileged Activity
⊡ The traditional way maybe…
□ File a ticket
□ Wait the ticket assign
□ Information collection
□ Approved by someone
□ Wait for operator change permission
□ Confirm you really get the permission
□ Start the task
□ Remove the permission by operator
41

42. 42

43. “
How About Achieving This
Efficiently Through ChatBot!
43

44. Change Management
W/ Chatbot
1. Submit RFC
2. File Ticket to Record3. Review By Stakeholder
5. Start to Perform the Change
4. Grant Permission
44

45. Using ChatBot
45

46. “Get All Security Information and
Auditing Function Ready, No Matter
From Cloud Provider or Third-Party
Solution
46

47. Alert and Notify of
Suspicious Activity
⊡ Only having enough data, then the security
checks and monitoring can happen
⊡ Setup basic rule set to monitor abnormal
behavior
⊡ Using AI to train the rule set
47

48. Suspicious Activity
⊡ Strange Login Behavior
⊡ Abnormal Inbound/Outbound Connection
⊡ Attack Event
⊡ Access Malicious Website
⊡ ...etc
48

49. “
How do You Patch Vulnerability?
49

50. Patch Vulnerability
Actively
50

51. “
More Secure CI/CD Flow - GitOps
51

52. GitOps
WebHook GitOps
Push
Push
Push
Push
Like Agent
Pulling
More Network Settings, Credentials
52

53. C.A.S. C.A.M.S.
5
53

54. DevOps Core Value
54

55. Culture
⊡ Except information security department,
everyone should have security knowledge
55

56. Automate
⊡ Penetration test integrate with release
pipeline
⊡ IaC avoid misconfiguration
⊡ Secret management rotate credentials
⊡ Chatbot
⊡ ...
56

57. Measure
⊡ Continuous security testing measure how
your service health
⊡ Rich system log measure how your system
safe
57

58. Share
⊡ Internal sharing makes vulnerability can be
excluded
⊡ External sharing make new patch
implemented quickly, incident decrease
58

59. THANKS!
Any questions?
You can find me at
facebook.com/smalltown0110
59