Recommended
More Related Content
Similar to CrikeyCon VI - The Boring Security Talk
Similar to CrikeyCon VI - The Boring Security Talk (20)
More from kieranjacobsen
More from kieranjacobsen (19)
Recently uploaded
Recently uploaded (20)
CrikeyCon VI - The Boring Security Talk
- 2. Hello! I am Kieran Jacobsen Head of Information Technology @ Readify Microsoft MVP, Cloud and Datacenter Management You can find me at: ◇ @kjacobsen ◇ Poshsecurity.com
- 3. CI/CD Pushing code around and around and around and around and …
- 4. Publicly Exposed ◇ Internet accessible ◇ Limited or no firewall rules
- 5. Weak Authentication ◇ No SSL/TLS ◇ Shared accounts ◇ Stale accounts ◇ No MFA
- 6. Significant Privileges ◇ Operating System privileges ◇ Cloud privileges
- 7. Patching ◇ Operating System ◇ CI/CD Tools ◇ Dependencies (Git)
- 8. Attacks Happen “Hackers exploit Jenkins servers, make $3 million by mining Monero”, CSO Online, 2018-02-20.
- 9. Restricting Access ◇ Does it need Internet access? ◇ Can we lock down by source IP address? ◇ Can we lock down to specific destination port numbers?
- 10. Using SSO and MFA ◇ Enable and enforce HTTPS ◇ Enable SSO – Each user has an account ◇ MFA should be enabled for Internet exposed systems
- 11. Least Privilege ◇ Ensure CI/CD agents and processes run with least privilege as possible ◇ Restrict who has admin access to CI/CD ◇ Audit privileges regularly
- 12. Patching ◇ Ensure servers are in regular patching process ◇ Plan for CI/CD patching and dependency tool patching
- 13. PR Validation
- 14. DNS The Internet street directory.
- 15. Change Control ◇ Who made a change? ◇ When did they make the change? ◇ Why did they do it? ◇ What was it pointing to?
- 16. Speed ◇ How long does it take to make a change? ◇ Manual changes
- 17. Visibility ◇ Do those impacted have visibility into changes?
- 18. Bad GUIs ◇ Cut-and-past errors ◇ Confusing terminology
- 19. Attacks Happen ◇ “DHS: Multiple US gov domains hit in serious DNS hijacking wave”, Ars Technica, 2019-01-26 ◇ “Advice on Mitigating DNS Infrastructure Tampering”, Posh Security, 2019-02-12 ◇ “A Deep Dive on the Recent Widespread DNS Hijacking Attacks”, Krebs on Security, 2019-02-18 ◇ “DNS Squatting with Azure App Services”, Posh Security, 2017-08-27
- 20. DNS Control ◇ Open Source Software ◇ Developed and maintained by Stack Overflow ◇ Supports multiple registrars and DNS providers ◇ Can preview changes before pushing them ◇ https://stackexchange.github.io/dnscontrol/
- 21. A Recognisable Format ◇ JavaScript configuration file ◇ Comments to help describe zone contents ◇ Example: http://bit.ly/dnsconfig
- 22. Version Control ◇ Branches ◇ Log ◇ Blame
- 23. Pull Requests ◇ Review changes ◇ Include impacted teams
- 24. CI/CD ◇ Humans don’t change DNS ◇ CI = DNSControl Preview ◇ CD = DNSControl Push ◇ Guide: http://bit.ly/DNSControl
- 28. Email The service we all have but don’t want
- 29. The Issues With Email ◇ SPAM ◇ Phishing ◇ Spear Phishing ◇ Whaling ◇ Impersonation
- 31. From: "Have I Been Pwned“ <noreply@haveibeenpwned.com> Return-Path: bounces+3489673- b289- myuser=mydomain.com@mail.haveibe enpwned.com
- 32. Identifying Sources ◇ Your mail servers ◇ Applications ◇ Marketing campaign servers ◇ Bulk email services ◇ SaaS products
- 33. “ We can’t use SPF, DKIM or DMARC because we don’t know who is legitimately sending email as our organisation‽
- 34. SPF ◇ Validates mail is coming from authorised IP addresses ◇ Information stored in DNS ◇ Validates envelope-from address ◇ Can include other SPF records – Office 365 etc ◇ DNS query limitations
- 35. DKIM ◇ Uses digital signatures to validate mail ◇ Validates Message-From header address ◇ Public key(s) stored in DNS
- 36. DMARC ◇ SPF and/or DKIM ◇ Alignment checks ◇ Allows domains to specify action if checks fail ◇ Reporting ◇ Policy stored in DNS
- 38. Packages & Dependencies Turtles all the way down
- 39. Browsealoud “UK ICO, USCourts.gov... Thousands of websites hijacked by hidden crypto-mining code after popular plugin pwned”, The Register, 2018-02-11
- 40. Docker Images “Malicious Docker Containers Earn Cryptomining Criminals $90K”, Threat Post, 2018-06-13
- 41. Eslint “Postmortem for Malicious Packages Published on July 12th, 2018”, ESLint, 2018-07
- 42. “ The maintainer whose account was compromised had reused their npm password on several other sites and did not have two-factor authentication enabled on their npm account.
- 43. Bootstrap-sass “Malicious remote code execution backdoor discovered in the popular bootstrap-sass Ruby gem”, Snyk Blog, 2019-04-03
- 44. Pipdig Power Pack (P3) “Pipdig Update: Dishonest Denials, Erased Evidence and Ongoing Offences”, Wordfence Blog, 2019-04-02
- 45. Thanks! Any questions? You can find me at: ◇ @kjacobsen ◇ Poshsecurity.com
Editor's Notes
- Good afternoon everyone, my name is Kieran Jacobsen, this is my fourth time speaking at CrikeyCon. It is such a pleasure to be returning to speak to you all this year. Last time I was at CrikeyCon I spoke about DevSecOps. That content was built upon my experiences with DevOps and DevSecOps. Today’s content is an extension of that in a lot of respects. I want to talk about topics that are often overlooked, protecting CI/CD infrastructure, email security practices, DNS management and understanding what goes on when dependencies go bad.
- Let’s start with our continuous integration and delivery systems. In a DevOps world, these are at the heart of a smooth running organisation. They are mission critical. Has anyone here ever considered the chaos an attacker could get up to if they pwned this infrastructure? Let’s look at some common issues.
- People love to overshare on the Internet. It is common to see start-ups and smaller organisations operating entirely in the cloud and exposing management interfaces and API endpoints to the internet. This isn’t SaaS offerings like Azure DevOps or GitLab, this is TeamCity and Jenkins instances.
- There are quite a few common authentication mistakes. The lack of HTTPS is a major issue, however it isn’t as likely to get you owned. The use of shared accounts or generic accounts is very common. This isn’t just a licensing violation, but you are destroying your audit trail. You have no idea who is pushing code into production. When named accounts are being used, stale accounts are common. Systems where developers who are long gone still have access. There are endless stories about disgruntled former employees. The lack of MFA is another issue. MFA is one of the strongest defences against credential spraying attacks.
- I am often contacted by consultants who are working in heavily locked down networks. Developers don’t have administrative rights on production, in some cases they don’t in development environments or their own workstations. The security teams are happy, everything is locked down. What they haven’t realised is that there is a large hole in this model. Devs have admin rights in the CI/CD systems, and these have administrator rights to prod. The developers actually have admin rights in prod. How does this happen? Cloud access accounts and keys are often given far to much permissions, for instance AWS keys with access to far too many resources and actions.
- How do I put this in a family friend way. I really don’t understand why patching is hard. Most organisations have OS patching plans, but are then running old version of their CI/CD tools, git and their compilers. Jenkins had a rough year last year. If you have it running in your organisation, how sure are you all of your instances are updated?
- In February last year, hackers exploited vulnerabilities in Jenkins to make over 3 million dollars by mining Monero. They were successful for the sheer fact that so many Jenkins servers are exposed to the internet and from experience, they have a lot of processing power. What shocks me is that no one seemed to notice their builds or deployments taking longer. Would your AWS or Azure bill be the only way that you would detect these attacks? So how do we protect against these attacks?
- Network security 101 tells us that we should be restricting access where possible. Do you need to expose your CI/Cd systems to the internet? If all of your development and production systems reside within your corporate network, then your CI/CD probably doesn’t need to be Internet accessible. If it does need to be accessible, can you restrict access to a set of trusted IP addresses? Perhaps a set of IP addresses you know legitimate connections will come from? If you can’t do that, then look at restricting what destination ports we can connect to. For instance, if the management interface is on HTTPS, and we use RDP or SSH to manage the server, then we really don’t need to make any other services like SMB Internet accessible.
- Authentication and privileged access should not occur over plaintext protocols like HTTP. Certificates are very easy to obtain and don’t be lazy and use self-signed certificates. Users should have their own accounts. No shared accounts. Personal preference is that these accounts be connected via Single Sign-On. I don’t care if it is Azure AD, AD, Open LDAP or Google Apps, just use a centralised identity provider. The benefits of SSO are better auditing, monitoring and user account deprovisioning. If you are exposing your CI/CD system to the Internet, then you must use multi-factor authentication. My opinion is that this is mandatory requirement. Side comment here. If you are an application vendor who doesn’t support SSO or MFA in 2019, or want to charge me for the right, I am going to look at another vendor. You obviously don’t care about security.
- Your CI/CD agents and service accounts should use the least privileges as possible. Avoid using local system in Windows and root on Linux. If you are running within a Active Directory domain, do not use a domain admin account for CI/CD. Restrict your AWS keys to only the resources and actions you need to perform, and assign access to Azure at a resource or resource group level, not at a subscription level. Restrict who has administrative access to your CI/CD. If a junior developer only needs to see the status of builds or deployments, then they should only have the privileges to do just that. Finally review everything on a regular basis.
- Build a patching plan that includes all of the parts of your developer tool chain. IDEs, developer tools, compilers, packaging software and your CI/CD platforms. You probably all have OS patching plans, include the rest of these in that same process.
- Before I finish talking about CI/CD, I quickly wanted to talk about an interesting story I saw from 2017. There were bot that would hunt for projects using CI for PR validation. These bots would replace the code of the project with bitcoin mining code that would then be automatically run as part of the PR validation process. GitHub put controls in place to prevent this from occurring, but I want you to think about if you could detect this type of attack, but it is an interesting detection story.
- DNS is one of those critical systems that never gets the attention it deserves. Every organization relies on DNS, yet, it's rarely considered as important as printing and email. Let’s take a look at some of the challenges we have all seen with DNS Management.
- I would say that in my experience, organizations struggle in understanding what DNS entries they have, why they are there and when they were created or modified. Was that entry added by one of your system administrators or an external attacker? Now if you work in an environment where there are change management processes like ITIL, you might be feeling a bit smug. I have two challenges for you. Do you think your change process captures enough information? Second, how much time would it take you to find the appropriate change request for an entry I randomly point to in your DNS zone? Would it take 5 minutes? 30? An hour? A day?
- The next challenge is the speed at which DNS changes are being made. In the past, DNS changes happened infrequently. A system engineer or network engineer would perform DNS changes as part of a once or twice a year application upgrade cycle. Records would often remain unchanged for years at a time. Things are changing. We want faster application release cycles, we want daily releases, we want DevOps, yet most teams haven’t changed how they approach DNS. Changes are still being performed manually by someone in the operations team.
- If only I had a dollar for every outage caused by DNS, I would probably wouldn’t need to work! Why do these issues occur? Often it is a simple issue of visibility, those impacted by a DNS change weren’t aware it was occurring. Change Management should have helped with this, yet from experience, it often hides these changes.
- With the move to providers like CloudFlare, Azure and Route53, more organizations are outsourcing the hosting of their external DNS. This has improved uptime but introduces another source of problems. The main mechanism for management is often a web portal, and even with APIs I see a lot of teams drifting towards these GUIs. The biggest issue I see is transposition errors, that is, copy and past errors leading to incorrect DNS entries. Confusing terminology and no standardization is also a contributing factor to DNS configuration errors. We need a method of working with DNS that is standardized across providers.
- Attacks against DNS have become a bit more popular in the last few months. In January there were reports of multiple US government domains being hit by DNS hijacking, and things going so far as DHS and GCHQ releasing emergency directives warning about the risks. These two organisations don’t release directives very often, in fact for DHS this was their first emergency directive. Dangling DNS entries are a major concern, particularly where public cloud services are in use. Situations where DNS records, specifically CNAME records are creating pointing to cloud services. At a later date, these cloud services are deleted but those CNAME records remain. They are left dangling. This happened to Microsoft in 2017. A research at Vulnerability lab discovered that resnet.Microsoft.com was a CNAME entry pointing to resnetportal-prod.azurewebsites.net. The only issue was that the website had been deleted. All an attacker would need to do is create a new app service with the address of resnetportal-prod.azurewebsites.net, and they would be able to take over the resnet.microsoft.com sub domain. Can you imagine the effectiveness of a phishing attack using a page hosted on a Microsoft subdomain? So, what tools and processes can we use to ensure we have streamlined DNS management?
- DNSControl is open source, written in GO and maintained by the Stack Overflow team. It allows you to preview changes before pushing them to your DNS provider. At Readify, we have been using DNSControl to manage our own domains for a bit over a year. Anyone in Readify can propose a DNS change, and our Platforms team can then review and approve those requests. This has empowered developers to propose changes and removed my team as a bottle neck for new projects. So how does DNS Control help us solve some of our DNS management problems?
- DNSControl uses JavaScript to define We need to write our DNS zone in a simple format that can be understood by everyone. The format should be easy enough for anyone to propose, review and understand changes. With DNSControl, a zone is defined using JavaScript. You don’t need to be a JavaScript expert to make changes, I have no idea how to write JavaScript and haven’t had any issues. The screenshot is a small piece of the Planet PowerShell DNS zone, see how easily you can tell which records are A, TXT and CNAME entries. Planet PowerShell is a community PowerShell content aggregator I run. If you are interested, you can see this entire file up on GitHub. Within the human readable format, we should be able to use comments to describe the contents of the zone. Comments allow us to answer the age-old question: “what’s this record for?”. We can also use comments to explain decisions within the zone.
- Version control systems, like GIT, provide a core piece of this DNS puzzle. We store out DNSControl files in GIT as it gives us a history of all changes to our zone and who made those changes. Git branches are cheap and easy to merge, allowing for developers and engineers to make changes, verify them and them push them into production. Git becomes our change tracking tool. With Git Log and Git Blame, we can easily determine what changes were made recently and by who.
- Platforms like GitHub and Azure DevOps extend Git with Pull Requests. If you are not familiar with pull requests, they are much like change requests in ITIL, but they are easier to create, review and don’t need a meeting to have them approved. Pull requests shine a light on changes in a way that ITIL was promised to organizations but didn’t deliver. We can include teams and individuals who are required review a PR that represent groups who may be impacted. A change to MX records might require approval from email administrators and the security team.
- The use of CI/CD is the final piece of puzzle. This reduces the risks of human generated errors. With DNSControl, we can run our CI process for all pull requests. We use DNSControl’s preview function to validate the changes and report what actions it would take. This allows those reviewing the change to have confidence in the changes being made. Next, once a PR is approved and the changes merged into master, our CD process can complete. This process will perform the push commands and actually perform the changes. As you can see, we have a process for managing DNS that has change tracking, better visibility and is automated that doesn’t require sharing credentials with every developer and sysadmin in the company.
- Before I finish up on DNS, lets look at one more example. Here you can see a page on Dell’s website, it is talking about their hardware-based encryption. The page has links to white papers and case studies to encourage you to use their product. Let’s assume I am interested in the Trusted Computers guide that is in the bottom left corner. Let’s hover over the link to see where it goes.
- That is a weird address. Dell4slg.com? What is that? Dell4SLG stands for Dell for State and Local Government. It was a website marketing Dell products and services towards those groups, and quite a bit of content was at one point hosted there. What happens when I click on this link?
- That’s not the Dell website, that is a plumbing supplies company in Turkey! It seems some Internet prankster has noticed the domain wasn’t renewed and decided to have some fun and setup a fun little traffic redirector. Your domains and their associated entries are part of your organisations brand. You need to take steps to protect these as you would any other brand asset.
- As an industry, we like to believe that the world is moving away from email. The truth is, even if you use Slack for al your internal communications, email will be used to communicate with your customers and suppliers. We also use email for activities like account confirmations and password resets. Considering how sensitive email is to our business, it doesn’t appear that many businesses take the steps to ensure that recipients of our emails can verify them as legitimate.
- Where do we start with what is wrong with email. Pretty much all email based attacks are still increasing and the quality is improving year on year. There are more stories of whaling style attacks against executives and high-profile targets. These emails come in the form of legal subpoenas or customer complaints, they are concerns that hook executives very effectively. I think we have probably all heard about the various executive impersonation attacks. Attackers pose as senior executives and send requests to finance or admin teams or executive assistants. There is a request for an urgent money transfer to complete a deal or purchasing gift cards to secure a client. These attacks create a sense of urgency and posing as the CEO results in requests being rarely questions. My frustration with all of this is that we can make these attacks harder, but most organisations seem hesitant to put the controls in place. Before looking at these controls, lets touch on some email basics.
- Here is an email that has been sent by Have I Been Pwned. Every email message is made of two parts. Headers, which are key value pairs providing delivery and diagnostic information, and the message body, the bit we see in our mail client.
- These are the headers for that email. There are 60 headers for this message alone. IANA currently list 335 permanent and 56 provisional headers, however mail servers can add their own custom headers. In this example, more than half a specific to Office 365. Let’s take a look at 2 of the more important headers.
- The from header can be a source of confusion. The problem is that when we send a message using SMTP, we specify who the message is from twice. When the SMTP connection is built, the sending SMTP client specifies the sender’s email address; this is often called the envelope from or SMTP from. Later on, in the mail headers, the client will specify who the sender is again. This we call the message or header from field. It is this that mail clients use to display who the message is from. These two don’t have to match, they typically do, but in some cases, for instance when using platforms like Sendgrid, they may not match. The RETURN-PATH header is created when a server accepts the email message and contains the enveloper from.
- Now that we understand some basics, we can start to take control. The first step is to find out who is sending email. There’s the obvious sources, corporate mail servers and your internal applications, but there are other sources that your IT team might not know about. Your marketing, sales and development teams might be using services like Mailchip and SendGrid to send emails to your customers. A big hurdle is SaaS products. A number of these platforms will want to send as your domain, for instance, an employee creates leave request, the HR system will send an email to their manager as the employee requesting approval.
- In some organisations, the thought of determining all the sources is just too much. Most organisations have policies around their use of their brand. What colours can be used, how their logo can be used. Your domain is part of your brand. You should be protecting it as you would any other brand asset. If you don’t want to take steps to protect it, why should I believe you care about security? Don’t you want to record who is sending email as your brand?
- The next step is implementing SPF. SPF is a list of IP addresses that can send email as our domain. The list is maintained in our DNS zone. Receivers can check this list and reject unauthorised messages. A common misconception is that SPF validates the message from header, it actually validates the envelope from. SPF has some advantages and disadvantages. SPF allows us to link different records together. This removes the need for administrators to maintain lists when they use Office 365 of Google Apps. Here is also the major issue. There is a limitation on the number of DNS queries that can be performed. In some organisations, care will need to be taken to ensure you are below this limit. Once you are comfortable that your SPF record is correct, you can move onto DKIM.
- DKIM uses digital signatures to validate authorised servers. It differs from end-to-end digital signatures like PGP or SMIME and doesn’t provide the same level of protection. These signatures are not visible to the recipient, they are added to the headers by the sending server and then validated by the receiving server. Validation is performed by through the publishing of the server’s public keys in the senders DNS. Unlike SPF, DKIM validates the Message-From header. DKIM is going to take more effort to implement than SPF. You need to ensure that each system that is sending email as your domain supports DKIM. With one vendor, we had to wait 6 months for them to implement DKIM. When reviewing SaaS products, particularly ones that send email, I always confirm DKIM support. Once you have DKIM implemented, the next step is a DMARC policy.
- A lot of organisations are terrified that creating a DMARC policy might result in their emails not being delivered to customers. Yes, you can shoot yourself in the foot with DMARC, but with care, you can ensure successful delivery. DMARC is built upon SPF and DKIM, it also provides more validation that the message from and envelop from are aligned. The idea is that if the SPF and DKIM checks are successful, and the headers are aligned, then the message is valid. Domains can specify different policies, that is, the actions receivers should take when a message fails verification. Senders can also specifie if recievers should provide reports on accepted and failed messages. Your DMARC policy is stored within DNS, I suggest starting with the none policy them moving to quarantine and finally reject. There is an interesting quirk with Office 365. Office 365 handles the reject policy as it would the quarantine policy. It ignores the senders wishes and will deliver failed messages to the users junk mail. I have no idea why they did this.
- I want to finish up by talking about bank grade security, I realise that is an oxymoron. The site Phishing Score card tracks the SPF, DKIM and DMARC configuration of organisations world wide. 53 percent of banks and financial service providers here in Australia have no dmarc configuration. 38 percent have a policy of none, and just 9 percent have either a reject or quarantine policy. How much do they really care about the security and privacy of their customers?
- Almost all software projects depend upon other software. This could be JavaScript packages, Docker images or even libraries provided by software vendors. The problem is, that we place a considerable amount of trust in the third-parties that write these dependencies. We often like to think that they are better people than we are. We like to believe that they will not introduce malicious code, and follow similar security practices as we do. Unfortunately, that trust gets broken. In the last few minutes I want to look at some of these incidents.
- February last year, the Javascript plugin, Browsealoud was compromised. The plugin’s source code was altered to inject Coinhive’s Monero miner. For several hours, anyone visiting the over 4 thousand impacted websites inadvertently ran the hidden mining code on their computer. Australian websites impacted included a number of government departments, city councils, charities and disability support service providers. SRI or Sub Resource Integrity is the best defence here. SRI allows a website to tell a users browser to check the integrity of third party assets. If the check fails, the asset isn’t loaded.
- In June last year, it was reported that 17 malicious Docker containers were removed from Docker Hub. Some of these images were mining Monero, while others created reverse shells or attempted to install backdoors onto the host machine. The images were available for over a year before Docker finally removed them. Estimates are that attackers made around 90 thousand from the mining. How do we protect against this? Well Sarah yesterday covered off this very eloquently, private image repositories is one of the best defences.
- Finally in July, an attacker compromised the NPM account of a maintainer of the popular ESLINT-SCOPE and ESLINT-CONFIG-ESLINT packages. They then published malicious versions of these packages that would send the contents of the user’s dot npmrc file to the attacker. This file typically contains access tokens for publishing to NPM. The belief is that they attackers wanted to gain access to the accounts of other package maintainers. These packages see around three million downloads each week and have 31 other packages directly depending upon them, and over 20 thousand indirectly depending upon them.
- How did the compromise occur? The official story is that a maintainer reused passwords and didn’t enable multi-factor authentication. As I said earlier, not all maintainers follow the same security practices as the rest of us.
- Just last week, attackers added a backdoor to the popular bootstrap-sass ruby gem. The great news about this one was that it took only a few hours between attackers posting the malicious version and it being detected and removed, unfortunately it was still downloaded around 12 thousand times. We don’t know how this one happened yet.
- Also last week, Wordpress plugin developers Pipdig were accused of adding a number of suspicious or malicious features, including remote killswitch, remote password changes and DDoSing a competitors website. It appears they even talk active measures to erase the evidence when caught. As I said, people trust these autors not to be malicious, and sometimes it turns out the author’s are quite malicious.
- I want to thank you all for listening to me today. I hope that you found something useful in today’s session and that it wasn’t too boring for you. I hope you enjoy the rest of CrikeyCon. Thank you all very much. <pause for applause> I believe we still have some time for questions, who has a question?