SlideShare a Scribd company logo
1 of 28
Download to read offline
Detecting secrets in code committed to Gitlab
(in real time)
Chandrapal Badshah
About Me
● Chandrapal Badshah
● Security Engineer
● Stoic and spends time with philosophy
● Pentest, Automation, Read books
● Manage @HackwithGithub on Twitter
Context
● Product based company, fail fast learn fast
● Hires a lot of devs*
● Use Gitlab community edition for code storage and CI/CD
● We do audit the code for secrets in regular intervals, but that’s late
Problem Statement
Need to detect and remove sensitive API keys (secrets) from code
This would reduce the impact when:
● Devs makes an internal repo public
● Devs pushes commits to their personal Github repos by mistake
● Unauthorized members accesses to code (insider threat)
This would help us in situations like
Source : https://www.bleepingcomputer.com/news/security/microsofts-github-account-hacked-private-repositories-stolen/
Let’s begin our journey
Git flow
→ git commit → git push →
Git hooks
● Git hooks are scripts that git executes before or after events such as:
commit, push, and receive
● Git hooks are a built-in feature - no need to download anything.
● There are many types of git hooks. Check out https://githooks.com/
● We are interested in commit and receive based hooks:
○ pre-commit
○ post-commit
○ pre-receive
○ post-receive
Git hooks in the flow
Source: https://blog.gitguardian.com/git-hooks-automated-secrets-detection/
Comparison of Git hooks
Pre commit and Post commit hooks - runs the scripts on dev machines.
Advantages:
● Stops even before the secrets are committed
Disadvantages:
● Adding new regex & managing the script on dev machines is hard
● False positives are bad user experience
● Privacy issues ? Nothing stops them from removing the git hooks
Comparison of Git hooks
Pre receive hook - it can’t do much checks as the code is yet to reach the server.
There is Pre push hook which executes even before the Pre receive hook is
executed on the server side. But Pre push hook is still on the client side.
Comparison of Git hooks
Post receive hook - runs on the server side.
Advantages:
● Can be configured for no delay when user does a git push. Devs don’t really
see the difference.
● Easy to manage the scripts
● False positives are manageable
Disadvantages:
● The secrets are already on the server
Final Decision
Go with the use of post receive hooks.
If secret detected:
● automatically raise a confidential Gitlab issue in the repo
● get feedback - check if it’s a false positive
● if it’s a secret, ask the devs to rotate the secret
Post receive hooks should be configured per repository
Gitlab feature to help post receive hooks
● Gitlab has System hooks
● Gitlab system hooks does a HTTP POST request for many events like push,
group create, repo create, etc
● More details at
https://docs.gitlab.com/ee/system_hooks/system_hooks.html
Existing secret detection tools
There are lots of open source tools:
● truffleHog
● gitleaks
● git-secrets by AWS Labs
● detect-secrets by Yelp
● talisman by ThoughtWorks
● and more...
TruffleHog
● Python based tool
● Customizable regex
● Easy install and CLI commands
● Good documentation
● https://github.com/dxa4481/truffleHog
Gitleaks
● Written in Golang
● Customizable regex
● Supports whitelisting of secrets
● Lots of options in CLI commands, lacks documentation
● Allows scan of single commit but downloads the entire repo
● https://github.com/zricethezav/gitleaks
Comparison of truffleHog and gitleaks
truffleHog
1. Efficient for smaller commits
2. Less memory intense
3. After configuring with Gitlab system hooks,
the total time taken to complete scanning
was less.
gitleaks
1. Same time as trufflehog for smaller commits.
Comparatively fast for huge commits.
2. Very greedy for CPU memory
3. After configuring with Gitlab system hooks,
the total time taken to complete scanning
was less but at the cost of CPU memory.
Changes made
● Took all the necessary code from truffleHog and stripped the rest. We
internally call it “tattletale-rt”.
● The scan logic looks like the below:
○ Get the code changes in the commit (only the added content not the removed)
○ Get all the regexes we need to scan
○ For each line in the code change, check if the regex matches
○ If matches, report it
● Have a separate service called “Issue Manager” which manages issues.
Final architecture
DEMO
Thanks to
Fahri Shihab
@fahrishb
Sanjog Panda
@sanjogpanda
What we learnt
● Not all API keys are sensitive. Google API keys are everywhere and are
intended to be public - Google Maps API key, Firebase key, etc
● Deployments are different for each projects - No “one solution” that fits all
● This detection is regex based. API keys / secrets will not be detected if:
○ API key doesn't match the regex
○ If the secrets are in a different language. пароль (parol’) is “password” in Russian.
● Entropy based detection is noisy but can detect some secrets.
● Learn on what’s the secure way to store secrets for each tech stack.
Thank you
Any Questions
What are we working on now ?
Follow on Twitter to get more updates on:
● Mobile App Security Pipeline (Android & iOS)
● SAST

More Related Content

What's hot

DevSecOps and the CI/CD Pipeline
 DevSecOps and the CI/CD Pipeline DevSecOps and the CI/CD Pipeline
DevSecOps and the CI/CD PipelineJames Wickett
 
Infrastructure as Code with Ansible
Infrastructure as Code with AnsibleInfrastructure as Code with Ansible
Infrastructure as Code with AnsibleDaniel Bezerra
 
GitHub Actions in action
GitHub Actions in actionGitHub Actions in action
GitHub Actions in actionOleksii Holub
 
CI and CD with Jenkins
CI and CD with JenkinsCI and CD with Jenkins
CI and CD with JenkinsMartin Málek
 
GitLab for CI/CD process
GitLab for CI/CD processGitLab for CI/CD process
GitLab for CI/CD processHYS Enterprise
 
CI/CD Overview
CI/CD OverviewCI/CD Overview
CI/CD OverviewAn Nguyen
 
Intro to Github Actions @likecoin
Intro to Github Actions @likecoinIntro to Github Actions @likecoin
Intro to Github Actions @likecoinWilliam Chong
 
SAST vs. DAST: What’s the Best Method For Application Security Testing?
SAST vs. DAST: What’s the Best Method For Application Security Testing?SAST vs. DAST: What’s the Best Method For Application Security Testing?
SAST vs. DAST: What’s the Best Method For Application Security Testing?Cigital
 
FOSDEM 2017: GitLab CI
FOSDEM 2017:  GitLab CIFOSDEM 2017:  GitLab CI
FOSDEM 2017: GitLab CIOlinData
 
Introduction to CI/CD
Introduction to CI/CDIntroduction to CI/CD
Introduction to CI/CDHoang Le
 
Azure DevOps & GitHub... Better Together!
Azure DevOps & GitHub... Better Together!Azure DevOps & GitHub... Better Together!
Azure DevOps & GitHub... Better Together!Lorenzo Barbieri
 
Introduction to Git and Github
Introduction to Git and GithubIntroduction to Git and Github
Introduction to Git and GithubHouari ZEGAI
 
CI/CD Best Practices for Your DevOps Journey
CI/CD Best  Practices for Your DevOps JourneyCI/CD Best  Practices for Your DevOps Journey
CI/CD Best Practices for Your DevOps JourneyDevOps.com
 

What's hot (20)

DEVSECOPS.pptx
DEVSECOPS.pptxDEVSECOPS.pptx
DEVSECOPS.pptx
 
DevSecOps and the CI/CD Pipeline
 DevSecOps and the CI/CD Pipeline DevSecOps and the CI/CD Pipeline
DevSecOps and the CI/CD Pipeline
 
Infrastructure as Code with Ansible
Infrastructure as Code with AnsibleInfrastructure as Code with Ansible
Infrastructure as Code with Ansible
 
GitHub Actions in action
GitHub Actions in actionGitHub Actions in action
GitHub Actions in action
 
CI and CD with Jenkins
CI and CD with JenkinsCI and CD with Jenkins
CI and CD with Jenkins
 
GitLab for CI/CD process
GitLab for CI/CD processGitLab for CI/CD process
GitLab for CI/CD process
 
Github in Action
Github in ActionGithub in Action
Github in Action
 
DevSecOps Jenkins Pipeline -Security
DevSecOps Jenkins Pipeline -SecurityDevSecOps Jenkins Pipeline -Security
DevSecOps Jenkins Pipeline -Security
 
Git
GitGit
Git
 
CI/CD with GitHub Actions
CI/CD with GitHub ActionsCI/CD with GitHub Actions
CI/CD with GitHub Actions
 
CI/CD Overview
CI/CD OverviewCI/CD Overview
CI/CD Overview
 
Intro to Github Actions @likecoin
Intro to Github Actions @likecoinIntro to Github Actions @likecoin
Intro to Github Actions @likecoin
 
SAST vs. DAST: What’s the Best Method For Application Security Testing?
SAST vs. DAST: What’s the Best Method For Application Security Testing?SAST vs. DAST: What’s the Best Method For Application Security Testing?
SAST vs. DAST: What’s the Best Method For Application Security Testing?
 
Jenkins CI
Jenkins CIJenkins CI
Jenkins CI
 
FOSDEM 2017: GitLab CI
FOSDEM 2017:  GitLab CIFOSDEM 2017:  GitLab CI
FOSDEM 2017: GitLab CI
 
Introduction to CI/CD
Introduction to CI/CDIntroduction to CI/CD
Introduction to CI/CD
 
Azure DevOps & GitHub... Better Together!
Azure DevOps & GitHub... Better Together!Azure DevOps & GitHub... Better Together!
Azure DevOps & GitHub... Better Together!
 
Introduction to Git and Github
Introduction to Git and GithubIntroduction to Git and Github
Introduction to Git and Github
 
CI/CD Best Practices for Your DevOps Journey
CI/CD Best  Practices for Your DevOps JourneyCI/CD Best  Practices for Your DevOps Journey
CI/CD Best Practices for Your DevOps Journey
 
Git commands
Git commandsGit commands
Git commands
 

Similar to Detecting secrets in code committed to gitlab (in real time)

GitLab Commit DevOps: How GitLab Can Save your Kubernetes environment from Be...
GitLab Commit DevOps: How GitLab Can Save your Kubernetes environment from Be...GitLab Commit DevOps: How GitLab Can Save your Kubernetes environment from Be...
GitLab Commit DevOps: How GitLab Can Save your Kubernetes environment from Be...Nico Meisenzahl
 
Que nos espera a los ALM Dudes para el 2013?
Que nos espera a los ALM Dudes para el 2013?Que nos espera a los ALM Dudes para el 2013?
Que nos espera a los ALM Dudes para el 2013?Bruno Capuano
 
Engage 2018 adm04 - The lazy admin wins
Engage 2018   adm04 - The lazy admin winsEngage 2018   adm04 - The lazy admin wins
Engage 2018 adm04 - The lazy admin winsMatteo Bisi
 
Engage 2018 adm04 The lazy admin wins
Engage 2018   adm04 The lazy admin winsEngage 2018   adm04 The lazy admin wins
Engage 2018 adm04 The lazy admin winsFactor-y S.r.l.
 
Overview of Gitlab usage
Overview of Gitlab usageOverview of Gitlab usage
Overview of Gitlab usageOluDouglas
 
Infrastructure as Code Patterns
Infrastructure as Code PatternsInfrastructure as Code Patterns
Infrastructure as Code PatternsKris Buytaert
 
stackconf 2022: Infrastructure Automation (anti) patterns
stackconf 2022: Infrastructure Automation (anti) patternsstackconf 2022: Infrastructure Automation (anti) patterns
stackconf 2022: Infrastructure Automation (anti) patternsNETWAYS
 
Git for work groups ironhack talk
Git for work groups ironhack talkGit for work groups ironhack talk
Git for work groups ironhack talkTiago Ameller
 
Webinar: End to End Security & Operations with Chainguard and Weave GitOps
Webinar: End to End Security & Operations with Chainguard and Weave GitOpsWebinar: End to End Security & Operations with Chainguard and Weave GitOps
Webinar: End to End Security & Operations with Chainguard and Weave GitOpsWeaveworks
 
My "Perfect" Toolchain Setup for Grails Projects
My "Perfect" Toolchain Setup for Grails ProjectsMy "Perfect" Toolchain Setup for Grails Projects
My "Perfect" Toolchain Setup for Grails ProjectsGR8Conf
 
Introduction to GitHub Actions
Introduction to GitHub ActionsIntroduction to GitHub Actions
Introduction to GitHub ActionsKnoldus Inc.
 
GitOps - Modern best practices for high velocity app dev using cloud native t...
GitOps - Modern best practices for high velocity app dev using cloud native t...GitOps - Modern best practices for high velocity app dev using cloud native t...
GitOps - Modern best practices for high velocity app dev using cloud native t...Weaveworks
 
Puppet Camp NYC 2014: Safely storing secrets and credentials in Git for use b...
Puppet Camp NYC 2014: Safely storing secrets and credentials in Git for use b...Puppet Camp NYC 2014: Safely storing secrets and credentials in Git for use b...
Puppet Camp NYC 2014: Safely storing secrets and credentials in Git for use b...Puppet
 
Open Sprintera (Where Open Source Sparks a Sprint of Possibilities)
Open Sprintera (Where Open Source Sparks a Sprint of Possibilities)Open Sprintera (Where Open Source Sparks a Sprint of Possibilities)
Open Sprintera (Where Open Source Sparks a Sprint of Possibilities)GDSCNiT
 
Hacking for fun & profit - The Kubernetes Way - Demi Ben-Ari - Panorays
Hacking for fun & profit - The Kubernetes Way - Demi Ben-Ari - PanoraysHacking for fun & profit - The Kubernetes Way - Demi Ben-Ari - Panorays
Hacking for fun & profit - The Kubernetes Way - Demi Ben-Ari - PanoraysDemi Ben-Ari
 
Continuous Security for GitOps
Continuous Security for GitOpsContinuous Security for GitOps
Continuous Security for GitOpsWeaveworks
 

Similar to Detecting secrets in code committed to gitlab (in real time) (20)

GitLab Commit DevOps: How GitLab Can Save your Kubernetes environment from Be...
GitLab Commit DevOps: How GitLab Can Save your Kubernetes environment from Be...GitLab Commit DevOps: How GitLab Can Save your Kubernetes environment from Be...
GitLab Commit DevOps: How GitLab Can Save your Kubernetes environment from Be...
 
Que nos espera a los ALM Dudes para el 2013?
Que nos espera a los ALM Dudes para el 2013?Que nos espera a los ALM Dudes para el 2013?
Que nos espera a los ALM Dudes para el 2013?
 
Engage 2018 adm04 - The lazy admin wins
Engage 2018   adm04 - The lazy admin winsEngage 2018   adm04 - The lazy admin wins
Engage 2018 adm04 - The lazy admin wins
 
Engage 2018 adm04 The lazy admin wins
Engage 2018   adm04 The lazy admin winsEngage 2018   adm04 The lazy admin wins
Engage 2018 adm04 The lazy admin wins
 
Overview of Gitlab usage
Overview of Gitlab usageOverview of Gitlab usage
Overview of Gitlab usage
 
Infrastructure as Code Patterns
Infrastructure as Code PatternsInfrastructure as Code Patterns
Infrastructure as Code Patterns
 
stackconf 2022: Infrastructure Automation (anti) patterns
stackconf 2022: Infrastructure Automation (anti) patternsstackconf 2022: Infrastructure Automation (anti) patterns
stackconf 2022: Infrastructure Automation (anti) patterns
 
Introduction to git & github
Introduction to git & githubIntroduction to git & github
Introduction to git & github
 
Git for work groups ironhack talk
Git for work groups ironhack talkGit for work groups ironhack talk
Git for work groups ironhack talk
 
Webinar: End to End Security & Operations with Chainguard and Weave GitOps
Webinar: End to End Security & Operations with Chainguard and Weave GitOpsWebinar: End to End Security & Operations with Chainguard and Weave GitOps
Webinar: End to End Security & Operations with Chainguard and Weave GitOps
 
Git Basics
Git BasicsGit Basics
Git Basics
 
My "Perfect" Toolchain Setup for Grails Projects
My "Perfect" Toolchain Setup for Grails ProjectsMy "Perfect" Toolchain Setup for Grails Projects
My "Perfect" Toolchain Setup for Grails Projects
 
Git sourcecontrolpreso
Git sourcecontrolpresoGit sourcecontrolpreso
Git sourcecontrolpreso
 
Introduction to GitHub Actions
Introduction to GitHub ActionsIntroduction to GitHub Actions
Introduction to GitHub Actions
 
GitOps - Modern best practices for high velocity app dev using cloud native t...
GitOps - Modern best practices for high velocity app dev using cloud native t...GitOps - Modern best practices for high velocity app dev using cloud native t...
GitOps - Modern best practices for high velocity app dev using cloud native t...
 
Puppet Camp NYC 2014: Safely storing secrets and credentials in Git for use b...
Puppet Camp NYC 2014: Safely storing secrets and credentials in Git for use b...Puppet Camp NYC 2014: Safely storing secrets and credentials in Git for use b...
Puppet Camp NYC 2014: Safely storing secrets and credentials in Git for use b...
 
Open Sprintera (Where Open Source Sparks a Sprint of Possibilities)
Open Sprintera (Where Open Source Sparks a Sprint of Possibilities)Open Sprintera (Where Open Source Sparks a Sprint of Possibilities)
Open Sprintera (Where Open Source Sparks a Sprint of Possibilities)
 
Hacking for fun & profit - The Kubernetes Way - Demi Ben-Ari - Panorays
Hacking for fun & profit - The Kubernetes Way - Demi Ben-Ari - PanoraysHacking for fun & profit - The Kubernetes Way - Demi Ben-Ari - Panorays
Hacking for fun & profit - The Kubernetes Way - Demi Ben-Ari - Panorays
 
DevOps Workshop Part 1
DevOps Workshop Part 1DevOps Workshop Part 1
DevOps Workshop Part 1
 
Continuous Security for GitOps
Continuous Security for GitOpsContinuous Security for GitOps
Continuous Security for GitOps
 

More from Chandrapal Badshah

Dangling DNS records takeover at scale
Dangling DNS records takeover at scaleDangling DNS records takeover at scale
Dangling DNS records takeover at scaleChandrapal Badshah
 
How to get started in InfoSec ?
How to get started in InfoSec ?How to get started in InfoSec ?
How to get started in InfoSec ?Chandrapal Badshah
 
OSINT mindset to protect your organization - Null monthly meet version
OSINT mindset to protect your organization - Null monthly meet versionOSINT mindset to protect your organization - Null monthly meet version
OSINT mindset to protect your organization - Null monthly meet versionChandrapal Badshah
 
OSINT Mindset to protect your Organization
OSINT Mindset to protect your OrganizationOSINT Mindset to protect your Organization
OSINT Mindset to protect your OrganizationChandrapal Badshah
 
Solving OWASP MSTG CrackMe using Frida
Solving OWASP MSTG CrackMe using FridaSolving OWASP MSTG CrackMe using Frida
Solving OWASP MSTG CrackMe using FridaChandrapal Badshah
 
Pentesting Android Apps using Frida (Beginners)
Pentesting Android Apps using Frida (Beginners)Pentesting Android Apps using Frida (Beginners)
Pentesting Android Apps using Frida (Beginners)Chandrapal Badshah
 
Let’s hunt the target using OSINT
Let’s hunt the target using OSINTLet’s hunt the target using OSINT
Let’s hunt the target using OSINTChandrapal Badshah
 

More from Chandrapal Badshah (11)

Dangling DNS records takeover at scale
Dangling DNS records takeover at scaleDangling DNS records takeover at scale
Dangling DNS records takeover at scale
 
How to get started in InfoSec ?
How to get started in InfoSec ?How to get started in InfoSec ?
How to get started in InfoSec ?
 
OSINT mindset to protect your organization - Null monthly meet version
OSINT mindset to protect your organization - Null monthly meet versionOSINT mindset to protect your organization - Null monthly meet version
OSINT mindset to protect your organization - Null monthly meet version
 
OSINT Mindset to protect your Organization
OSINT Mindset to protect your OrganizationOSINT Mindset to protect your Organization
OSINT Mindset to protect your Organization
 
Solving OWASP MSTG CrackMe using Frida
Solving OWASP MSTG CrackMe using FridaSolving OWASP MSTG CrackMe using Frida
Solving OWASP MSTG CrackMe using Frida
 
OWASP Serverless Top 10
OWASP Serverless Top 10OWASP Serverless Top 10
OWASP Serverless Top 10
 
Pentesting Android Apps using Frida (Beginners)
Pentesting Android Apps using Frida (Beginners)Pentesting Android Apps using Frida (Beginners)
Pentesting Android Apps using Frida (Beginners)
 
Let’s hunt the target using OSINT
Let’s hunt the target using OSINTLet’s hunt the target using OSINT
Let’s hunt the target using OSINT
 
pwnd.sh
pwnd.shpwnd.sh
pwnd.sh
 
Web Application Firewall
Web Application FirewallWeb Application Firewall
Web Application Firewall
 
Netcat - A Swiss Army Tool
Netcat - A Swiss Army ToolNetcat - A Swiss Army Tool
Netcat - A Swiss Army Tool
 

Recently uploaded

logical backup of Oracle Datapump-detailed.pptx
logical backup of Oracle Datapump-detailed.pptxlogical backup of Oracle Datapump-detailed.pptx
logical backup of Oracle Datapump-detailed.pptxRemote DBA Services
 
Keeping your build tool updated in a multi repository world
Keeping your build tool updated in a multi repository worldKeeping your build tool updated in a multi repository world
Keeping your build tool updated in a multi repository worldRoberto Pérez Alcolea
 
Reliable from-source builds (Qshare 28 Nov 2023).pdf
Reliable from-source builds (Qshare 28 Nov 2023).pdfReliable from-source builds (Qshare 28 Nov 2023).pdf
Reliable from-source builds (Qshare 28 Nov 2023).pdfRalf Gommers
 
What is Mendix and the concept of low-code development.docx
What is Mendix and the concept of low-code development.docxWhat is Mendix and the concept of low-code development.docx
What is Mendix and the concept of low-code development.docxTechnogeeks
 
Tech Tuesday Slides - Introduction to Project Management with OnePlan's Work ...
Tech Tuesday Slides - Introduction to Project Management with OnePlan's Work ...Tech Tuesday Slides - Introduction to Project Management with OnePlan's Work ...
Tech Tuesday Slides - Introduction to Project Management with OnePlan's Work ...OnePlan Solutions
 
ManageIQ - Sprint 234 Review - Slide Deck
ManageIQ - Sprint 234 Review - Slide DeckManageIQ - Sprint 234 Review - Slide Deck
ManageIQ - Sprint 234 Review - Slide DeckManageIQ
 
Zer0con 2024 final share short version.pdf
Zer0con 2024 final share short version.pdfZer0con 2024 final share short version.pdf
Zer0con 2024 final share short version.pdfmaor17
 
AmsterdamJUG April 2024 - Going serverless with Quarkus GraalVM native images...
AmsterdamJUG April 2024 - Going serverless with Quarkus GraalVM native images...AmsterdamJUG April 2024 - Going serverless with Quarkus GraalVM native images...
AmsterdamJUG April 2024 - Going serverless with Quarkus GraalVM native images...Bert Jan Schrijver
 
Effort Estimation Techniques used in Software Projects
Effort Estimation Techniques used in Software ProjectsEffort Estimation Techniques used in Software Projects
Effort Estimation Techniques used in Software ProjectsDEEPRAJ PATHAK
 
Leveraging the Expertise of a Social Media Fraud Analyst to Safeguard Brand R...
Leveraging the Expertise of a Social Media Fraud Analyst to Safeguard Brand R...Leveraging the Expertise of a Social Media Fraud Analyst to Safeguard Brand R...
Leveraging the Expertise of a Social Media Fraud Analyst to Safeguard Brand R...Milind Agarwal
 
Explore the Three Main Types of Logistics - Inbound Logistics, Outbound Logis...
Explore the Three Main Types of Logistics - Inbound Logistics, Outbound Logis...Explore the Three Main Types of Logistics - Inbound Logistics, Outbound Logis...
Explore the Three Main Types of Logistics - Inbound Logistics, Outbound Logis...Piyovi
 
Effectively Troubleshoot 9 Types of OutOfMemoryError
Effectively Troubleshoot 9 Types of OutOfMemoryErrorEffectively Troubleshoot 9 Types of OutOfMemoryError
Effectively Troubleshoot 9 Types of OutOfMemoryErrorTier1 app
 
ETE PPT.pdf LMMKLMKLMLKMLLMJKBHJBHBNUIHBU
ETE PPT.pdf LMMKLMKLMLKMLLMJKBHJBHBNUIHBUETE PPT.pdf LMMKLMKLMLKMLLMJKBHJBHBNUIHBU
ETE PPT.pdf LMMKLMKLMLKMLLMJKBHJBHBNUIHBUsamruddhijedgule2004
 
Business Analyzopedia - Your Pocket Gita for Business Analysis
Business Analyzopedia - Your Pocket Gita for Business AnalysisBusiness Analyzopedia - Your Pocket Gita for Business Analysis
Business Analyzopedia - Your Pocket Gita for Business AnalysisDEEPRAJ PATHAK
 
eSoftTools IMAP Backup Software and migration tools
eSoftTools IMAP Backup Software and migration toolseSoftTools IMAP Backup Software and migration tools
eSoftTools IMAP Backup Software and migration toolsosttopstonverter
 
What are the core components of Azure Data Engineer courses.docx
What are the core components of Azure Data Engineer courses.docxWhat are the core components of Azure Data Engineer courses.docx
What are the core components of Azure Data Engineer courses.docxkzayra69
 
Advantages of Cargo Cloud Solutions.pptx
Advantages of Cargo Cloud Solutions.pptxAdvantages of Cargo Cloud Solutions.pptx
Advantages of Cargo Cloud Solutions.pptxRTS corp
 
Understanding Plagiarism: Causes, Consequences and Prevention.pptx
Understanding Plagiarism: Causes, Consequences and Prevention.pptxUnderstanding Plagiarism: Causes, Consequences and Prevention.pptx
Understanding Plagiarism: Causes, Consequences and Prevention.pptxSasikiranMarri
 
oracle 23c new features for developer and dba
oracle 23c new features for developer and dbaoracle 23c new features for developer and dba
oracle 23c new features for developer and dbaRemote DBA Services
 
Santander Stream Processing with Apache Flink
Santander Stream Processing with Apache FlinkSantander Stream Processing with Apache Flink
Santander Stream Processing with Apache Flinkconfluent
 

Recently uploaded (20)

logical backup of Oracle Datapump-detailed.pptx
logical backup of Oracle Datapump-detailed.pptxlogical backup of Oracle Datapump-detailed.pptx
logical backup of Oracle Datapump-detailed.pptx
 
Keeping your build tool updated in a multi repository world
Keeping your build tool updated in a multi repository worldKeeping your build tool updated in a multi repository world
Keeping your build tool updated in a multi repository world
 
Reliable from-source builds (Qshare 28 Nov 2023).pdf
Reliable from-source builds (Qshare 28 Nov 2023).pdfReliable from-source builds (Qshare 28 Nov 2023).pdf
Reliable from-source builds (Qshare 28 Nov 2023).pdf
 
What is Mendix and the concept of low-code development.docx
What is Mendix and the concept of low-code development.docxWhat is Mendix and the concept of low-code development.docx
What is Mendix and the concept of low-code development.docx
 
Tech Tuesday Slides - Introduction to Project Management with OnePlan's Work ...
Tech Tuesday Slides - Introduction to Project Management with OnePlan's Work ...Tech Tuesday Slides - Introduction to Project Management with OnePlan's Work ...
Tech Tuesday Slides - Introduction to Project Management with OnePlan's Work ...
 
ManageIQ - Sprint 234 Review - Slide Deck
ManageIQ - Sprint 234 Review - Slide DeckManageIQ - Sprint 234 Review - Slide Deck
ManageIQ - Sprint 234 Review - Slide Deck
 
Zer0con 2024 final share short version.pdf
Zer0con 2024 final share short version.pdfZer0con 2024 final share short version.pdf
Zer0con 2024 final share short version.pdf
 
AmsterdamJUG April 2024 - Going serverless with Quarkus GraalVM native images...
AmsterdamJUG April 2024 - Going serverless with Quarkus GraalVM native images...AmsterdamJUG April 2024 - Going serverless with Quarkus GraalVM native images...
AmsterdamJUG April 2024 - Going serverless with Quarkus GraalVM native images...
 
Effort Estimation Techniques used in Software Projects
Effort Estimation Techniques used in Software ProjectsEffort Estimation Techniques used in Software Projects
Effort Estimation Techniques used in Software Projects
 
Leveraging the Expertise of a Social Media Fraud Analyst to Safeguard Brand R...
Leveraging the Expertise of a Social Media Fraud Analyst to Safeguard Brand R...Leveraging the Expertise of a Social Media Fraud Analyst to Safeguard Brand R...
Leveraging the Expertise of a Social Media Fraud Analyst to Safeguard Brand R...
 
Explore the Three Main Types of Logistics - Inbound Logistics, Outbound Logis...
Explore the Three Main Types of Logistics - Inbound Logistics, Outbound Logis...Explore the Three Main Types of Logistics - Inbound Logistics, Outbound Logis...
Explore the Three Main Types of Logistics - Inbound Logistics, Outbound Logis...
 
Effectively Troubleshoot 9 Types of OutOfMemoryError
Effectively Troubleshoot 9 Types of OutOfMemoryErrorEffectively Troubleshoot 9 Types of OutOfMemoryError
Effectively Troubleshoot 9 Types of OutOfMemoryError
 
ETE PPT.pdf LMMKLMKLMLKMLLMJKBHJBHBNUIHBU
ETE PPT.pdf LMMKLMKLMLKMLLMJKBHJBHBNUIHBUETE PPT.pdf LMMKLMKLMLKMLLMJKBHJBHBNUIHBU
ETE PPT.pdf LMMKLMKLMLKMLLMJKBHJBHBNUIHBU
 
Business Analyzopedia - Your Pocket Gita for Business Analysis
Business Analyzopedia - Your Pocket Gita for Business AnalysisBusiness Analyzopedia - Your Pocket Gita for Business Analysis
Business Analyzopedia - Your Pocket Gita for Business Analysis
 
eSoftTools IMAP Backup Software and migration tools
eSoftTools IMAP Backup Software and migration toolseSoftTools IMAP Backup Software and migration tools
eSoftTools IMAP Backup Software and migration tools
 
What are the core components of Azure Data Engineer courses.docx
What are the core components of Azure Data Engineer courses.docxWhat are the core components of Azure Data Engineer courses.docx
What are the core components of Azure Data Engineer courses.docx
 
Advantages of Cargo Cloud Solutions.pptx
Advantages of Cargo Cloud Solutions.pptxAdvantages of Cargo Cloud Solutions.pptx
Advantages of Cargo Cloud Solutions.pptx
 
Understanding Plagiarism: Causes, Consequences and Prevention.pptx
Understanding Plagiarism: Causes, Consequences and Prevention.pptxUnderstanding Plagiarism: Causes, Consequences and Prevention.pptx
Understanding Plagiarism: Causes, Consequences and Prevention.pptx
 
oracle 23c new features for developer and dba
oracle 23c new features for developer and dbaoracle 23c new features for developer and dba
oracle 23c new features for developer and dba
 
Santander Stream Processing with Apache Flink
Santander Stream Processing with Apache FlinkSantander Stream Processing with Apache Flink
Santander Stream Processing with Apache Flink
 

Detecting secrets in code committed to gitlab (in real time)

  • 1. Detecting secrets in code committed to Gitlab (in real time) Chandrapal Badshah
  • 2. About Me ● Chandrapal Badshah ● Security Engineer ● Stoic and spends time with philosophy ● Pentest, Automation, Read books ● Manage @HackwithGithub on Twitter
  • 3. Context ● Product based company, fail fast learn fast ● Hires a lot of devs* ● Use Gitlab community edition for code storage and CI/CD ● We do audit the code for secrets in regular intervals, but that’s late
  • 4. Problem Statement Need to detect and remove sensitive API keys (secrets) from code This would reduce the impact when: ● Devs makes an internal repo public ● Devs pushes commits to their personal Github repos by mistake ● Unauthorized members accesses to code (insider threat)
  • 5. This would help us in situations like Source : https://www.bleepingcomputer.com/news/security/microsofts-github-account-hacked-private-repositories-stolen/
  • 7. Git flow → git commit → git push →
  • 8. Git hooks ● Git hooks are scripts that git executes before or after events such as: commit, push, and receive ● Git hooks are a built-in feature - no need to download anything. ● There are many types of git hooks. Check out https://githooks.com/ ● We are interested in commit and receive based hooks: ○ pre-commit ○ post-commit ○ pre-receive ○ post-receive
  • 9. Git hooks in the flow Source: https://blog.gitguardian.com/git-hooks-automated-secrets-detection/
  • 10.
  • 11. Comparison of Git hooks Pre commit and Post commit hooks - runs the scripts on dev machines. Advantages: ● Stops even before the secrets are committed Disadvantages: ● Adding new regex & managing the script on dev machines is hard ● False positives are bad user experience ● Privacy issues ? Nothing stops them from removing the git hooks
  • 12. Comparison of Git hooks Pre receive hook - it can’t do much checks as the code is yet to reach the server. There is Pre push hook which executes even before the Pre receive hook is executed on the server side. But Pre push hook is still on the client side.
  • 13. Comparison of Git hooks Post receive hook - runs on the server side. Advantages: ● Can be configured for no delay when user does a git push. Devs don’t really see the difference. ● Easy to manage the scripts ● False positives are manageable Disadvantages: ● The secrets are already on the server
  • 14. Final Decision Go with the use of post receive hooks. If secret detected: ● automatically raise a confidential Gitlab issue in the repo ● get feedback - check if it’s a false positive ● if it’s a secret, ask the devs to rotate the secret Post receive hooks should be configured per repository
  • 15. Gitlab feature to help post receive hooks ● Gitlab has System hooks ● Gitlab system hooks does a HTTP POST request for many events like push, group create, repo create, etc ● More details at https://docs.gitlab.com/ee/system_hooks/system_hooks.html
  • 16.
  • 17. Existing secret detection tools There are lots of open source tools: ● truffleHog ● gitleaks ● git-secrets by AWS Labs ● detect-secrets by Yelp ● talisman by ThoughtWorks ● and more...
  • 18. TruffleHog ● Python based tool ● Customizable regex ● Easy install and CLI commands ● Good documentation ● https://github.com/dxa4481/truffleHog
  • 19. Gitleaks ● Written in Golang ● Customizable regex ● Supports whitelisting of secrets ● Lots of options in CLI commands, lacks documentation ● Allows scan of single commit but downloads the entire repo ● https://github.com/zricethezav/gitleaks
  • 20. Comparison of truffleHog and gitleaks truffleHog 1. Efficient for smaller commits 2. Less memory intense 3. After configuring with Gitlab system hooks, the total time taken to complete scanning was less. gitleaks 1. Same time as trufflehog for smaller commits. Comparatively fast for huge commits. 2. Very greedy for CPU memory 3. After configuring with Gitlab system hooks, the total time taken to complete scanning was less but at the cost of CPU memory.
  • 21. Changes made ● Took all the necessary code from truffleHog and stripped the rest. We internally call it “tattletale-rt”. ● The scan logic looks like the below: ○ Get the code changes in the commit (only the added content not the removed) ○ Get all the regexes we need to scan ○ For each line in the code change, check if the regex matches ○ If matches, report it ● Have a separate service called “Issue Manager” which manages issues.
  • 23. DEMO
  • 25.
  • 26. What we learnt ● Not all API keys are sensitive. Google API keys are everywhere and are intended to be public - Google Maps API key, Firebase key, etc ● Deployments are different for each projects - No “one solution” that fits all ● This detection is regex based. API keys / secrets will not be detected if: ○ API key doesn't match the regex ○ If the secrets are in a different language. пароль (parol’) is “password” in Russian. ● Entropy based detection is noisy but can detect some secrets. ● Learn on what’s the secure way to store secrets for each tech stack.
  • 28. What are we working on now ? Follow on Twitter to get more updates on: ● Mobile App Security Pipeline (Android & iOS) ● SAST