Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Meetup callback

40 views

Published on

Serverless Boston @ Veracode meetup
Bob Balaban presention on 21-June-2018

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Meetup callback

  1. 1. It’s More Ops Than You Think June 2018 Bob Balaban, CA/Veracode
  2. 2. Copyright © 2018 CA. All rights reserved. Agenda • The “promise” of serverless • Veracode Dynamic Analysis • Replacing a “server” with a “service” – Advantages – Dev and DevOps processes – Demos – Dev effort summary – Ops effort summary • Lessons Learned (including…security!) • Q & A
  3. 3. Copyright © 2018 CA. All rights reserved. Advantages of Serverless (from Wayne)  Automated Scalability  Built-in Fault Tolerance  LessOps – No infrastructure to manage  Lower Cost - Pay only for what you use  Faster development - Small functions easier to code and business logic focused  Lower Risk – responsibility transferred to provider  Integrated security – No patching, remote into O/S (SSH/RDP)  Automatic logging We will come back to this later
  4. 4. Copyright © 2018 CA. All rights reserved. Veracode Dynamic Analysis • Scans a “target app” (website) looking for security vulnerabilities – Cross-site scripting (XSS) – SQL Injection (SQLi) – XML External Entity (XXE) – Remote File Inclusion (RFI) – OS Command Injection – Server-side Request Forgery (SSRF) – Directory Traversal – …and many more • See OWASP: http://www.owasp.org
  5. 5. Copyright © 2018 CA. All rights reserved. How Do We Find Vulnerabilities? • “Ethical hacking” • We “attack” the target (in ways that don’t damage data or functionality) • If an attack succeeds, we report a vulnerability – And help you fix it! • Example: SQL Injection – “Select (sleep(15)) where true” – We can tell if it works because the response is sloooowwww
  6. 6. Copyright © 2018 CA. All rights reserved. What a “callback” server does 1. Scan engine attacks a target app with (ethically) malicious XML, Javascript, etc. 2. If successful, the attack causes the app to make a call to the Veracode “callback server” 3. The server records the successful attack 4. The scan engine polls the server later to fetch data on successful attacks, and reports vulnerabilities
  7. 7. Copyright © 2018 CA. All rights reserved. Callback Server (“Before”) Callback Server Target App Scan Engine 1. Inject attack, e.g., “include remote PHP file from callback service” 3. Get results 2. “I was hacked!”
  8. 8. Copyright © 2018 CA. All rights reserved. Existing Callback Server • AWS EC2, golang • Simple “GET” response to target apps – Serves static file – Records successful attacks • Simple REST API for scanner engine – Register/unregister scanners – Fetch successful attack data – Clean up attack data • Postgres RDB for attack data – Auto-purges after 7 days
  9. 9. Copyright © 2018 CA. All rights reserved. New Callback SERVICE Objectives Re-architect as “serverless”: 1. Lambda functions only run when invoked 2. AWS API Gateway offloads authorization, firewall and load balancing 3. DynamoDB nosql database easy to use from lambdas 4. Both Lambda and DynamoDB usage can auto-scale 5. Simplify REST API, otherwise maintain the same interfaces 6. Add some metrics tracking (count of invocations)
  10. 10. Copyright © 2018 CA. All rights reserved. New Callback Service (“After”) Callback Service Target App Scan Engine 1. Inject attack 3. Get results 2. “I was hacked!”
  11. 11. Copyright © 2018 CA. All rights reserved. Callback Service Components (AWS) DynamoDb Query attack info Capture attacks, Serve text A P I G a t e w a y From target app From scanner λ λ
  12. 12. Copyright © 2018 CA. All rights reserved. Tour of new Callback Service
  13. 13. Copyright © 2018 CA. All rights reserved. Advantages of New Architecture • DynamoDB TTL feature means we can simplify the service’s REST API – No need for cleanup calls – No need for Postgres • Lambdas do not get billed for idle time • API Gateway will auto-scale lambda invocations when needed (no need for auto-scaling groups) • Logging calls go automatically to CloudWatch
  14. 14. Copyright © 2018 CA. All rights reserved. Development Process Changes • Smaller, independent functions (good!) • No runtime debugging (bad!) – didn’t try Cloud9 – Fallback to “Printf debugging” – Also required to see what your function is being called with (e.g., http event data structure not documented) • Create AWS resources manually, the first time – API Gateway (can start with a Swagger definition) – Lambdas – IAM – DynamoDB tables – S3 (backup Terraform db file)
  15. 15. Copyright © 2018 CA. All rights reserved. Development Process Changes - 2 • Dev/test cycle – Make code changes – Manually (or semi-manually) deploy to AWS • Need to figure out how to get code from IDE to AWS • We use gitlab and Terraform • Terraform can be used locally to push changes to AWS • (Have to zip .js files first!) – Test (Postman/curl) – Commit changes to repo – Repeat
  16. 16. Copyright © 2018 CA. All rights reserved. Devops! • Manual deployment is not sustainable long-term • Must automate! • Chose Terraform to script AWS resource creation/configuration – Injectable “variables” for AWS keys, config parameters, environment variables – Resource dependencies (e.g., create lambdas before api gateway) – Incremental changes! (if you keep a copy of the tf db) • Chose Gitlab for CI/CD scripting – Triggered on merge to Master branch – Build zip files, manage Terraform invocation
  17. 17. Copyright © 2018 CA. All rights reserved. Devops - 2 • Ongoing re-factoring of the project – When and how to supply production account AWS keys – Coordinate Gitlab deploys and “legacy” (Bitbucket) builds – Who decides on gateway “stages”? • Who monitors the logs? – Some teams ship their logs to Sumo (more lambdas) • Who watches scalability of the gateway and lambdas? • Oh, and who watches cost?
  18. 18. Copyright © 2018 CA. All rights reserved. Time Commitment – Dev and DevOps • Dev was about 20% of “time to beta” • DevOps (pipeline automation, config automation…) was 80% • Why? – The pipeline automation tools were new to us, we had to learn them – Some AWS features were new
  19. 19. Copyright © 2018 CA. All rights reserved. Stuff We Had to Learn From Scratch • API Gateway setup/config • Lambda deployment/config – Permissions, environment variables • Querying DynamoDB efficiently • Terraform – create/update entire AWS infrastructure • Gitlab CI tools
  20. 20. Copyright © 2018 CA. All rights reserved. Skills We Had to Improve • Node.js – Coding – Debugging (via console.log() output, ew) • Cloudwatch • IAM for inter-component authorization – E.g., API Gateway calling Lambda – Lambda calling Cloudwatch and DynamoDB • PostMan/curl – Create http POST and GET tests
  21. 21. Copyright © 2018 CA. All rights reserved. Recap: Advantages of Serverless  Automated Scalability √  Built-in Fault Tolerance √  LessOps – No infrastructure to manage meh  Lower Cost - Pay only for what you use √  Faster development - Small functions easier to code and business logic focused √  Lower Risk – responsibility transferred to provider meh  Integrated security – No patching, remote into O/S (SSH/RDP) √  Automatic logging meh
  22. 22. Copyright © 2018 CA. All rights reserved. Lessons Learned • Serverless is great, where appropriate – Keep it simple • Figure out the DevOps as you go – Assume you will automate everything: • Build • Unit test • Deploy • Integration test • You will re-factor the project at least as often as you refactor your code
  23. 23. Copyright © 2018 CA. All rights reserved. And Oh Yeah, Security! “Security is like performance, you bolt it on at the end of the project”
  24. 24. Copyright © 2018 CA. All rights reserved. “Security is like performance, you bolt it on at the end of the project” (Said no one we want to listen to)
  25. 25. Copyright © 2018 CA. All rights reserved. Severless Security: Application Dependencies Application dependencies are going to become one of the main vectors for attackers to compromise your FaaS infrastructure. It is crucial that you scan third party libraries for vulnerabilities.  To drive home that point - a researcher recently found a way to get push rights to over 14% of NPM Packages and mainstream ones at that, e.g. : debug, request, react, gulp... How? It was a combination of brute force and credential leaks from GitHub.  In an evaluation of 1,000 open-source serverless projects, a threat research team revealed that 21% of them contained one or more critical vulnerabilities or misconfigurations, allowing attackers to manipulate applications and perform various malicious actions. Mitigations  Maintaining an inventory list of software packages and other dependencies and their versions  Scanning libraries for known vulnerable dependencies. Vulnerability scanning should be done as part of your ongoing CI/CD process
  26. 26. Copyright © 2018 CA. All rights reserved. Serverless Security: At-Rest and In-Transit equally vulnerable! Serverless exudes a false sense of security due to secure API calls and encrypted databases. 1. OWASP top 10 is still as relevant as ever! 2. SQL injection and other forms of injection attacks are still possible in the serverless world, as are cross-site scripting attacks. 3. Keep your secrets in an Key Management System, not code. 4. Ensure your services have the least amount of privilege to read/write to your databases. 5. Use direct/private connections between your VPCs and Databases wherever possible. Mitigations  Always use safe APIs that either sanitize or validate user input, or APIs which provide a mechanism for binding variables or parameterizing them for underlying infrastructure (e.g. stored procedures or prepared statements in the case of SQL queries)  Always use safe APIs that either sanitize or validate user input, or APIs which provide a mechanism for binding variables or parameterizing them for underlying infrastructure.
  27. 27. Copyright © 2018 CA. All rights reserved. Severless Security: Monitoring becomes extremely important The one thing that the advent of serverless has made more difficult is monitoring and the lack of understanding of security implications for various serverless assets.  Logging is key. Ensure you have CloudTrail enabled on all accounts for all regions. Don’t assume a developer or malicious actor won’t create an asset outside of us-east-1.  The days of agents are gone! You now have to use platform-specific tools to monitor…  …using Cloudwatch Metrics (with alarms) to monitor resources and set limits on either calls or spend. When alerted, investigate when/why those limits are being exceeded.  If you have a SIEM Tool or a 3rd-party monitoring service, send CloudWatch/Cloudtrail logs to their services to be monitored. You’ll be amazed at what is already actively scanning your public assets.  RYOM – (rolling your own monitoring) – Lambda’s written in Python using the Boto SDK are extremely powerful to monitor/modify your assets and alert based on customized parameters. e.g. - A developer adds their home ip address to connect to an asset for debugging from home, but a metric was defined to only allow connections of a specific CIDR range and kicked off the Lambda written with Boto to remove that home ip entry automatically from accessing that asset.
  28. 28. Copyright © 2018 CA. All rights reserved. Thank you! We’re hiring! Q & A

×