Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Multi-Account Strategy and Security with Centrica Hive

445 views

Published on

Multi-Account Strategy and Security with Centrica Hive

Multi-Account Strategy and Security with Centrica Hive

  1. 1. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Jay Harrison SRE Technical Lead - Centrica Hive Mark Davison SRE Security Consultant - Ronin IT Consulting for Centrica Hive AWS Multi Account Management & Security A new scaling challenge
  2. 2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. A brief history of Hive
  3. 3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. 2011 - British Gas Remote Heating Control Before Hive, British Gas developed RHC in partnership with AlertMe Ltd Ahead of its time but limited demand No significant device design improvement from older non-smart thermostats Moderately successful - 100k customers, mostly via British Gas upsell
  4. 4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. 2012 We started as British Gas Connected Homes 12 people in a borrowed basement office Building on lessons learned from the British Gas Remote Heating Control product
  5. 5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. 2013 First IoT product released Hive Active Heating v1 V1 Thermostat
  6. 6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. 2015 Hive Active Heating v2 Designed in conjunction with Yves Behar New features - improved UI and holiday mode V2 Thermostat
  7. 7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. 2016 Smart plugs, sensors, smart bulbs Hive Camera First diagnostic product - Boiler IQ Smart devices Hive Camera
  8. 8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. 2017 Second diagnostic product - Hive Leak Detector 3rd party services Amazon Echo, IFTTT, Google Home, Philips Hue Hive Leak Boiler IQ
  9. 9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. 2018 Hive View Camera Better features including event detection and event history Hive View
  10. 10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. SRE at Centrica Hive
  11. 11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. SRE at Hive We run the systems, services and tools used by our product development teams Service Engineers Building & maintaining development and product tools and services Product Engineers Using and refining the services. Embedded in the product teams Security Engineers Writing tools to secure the infrastructure, users and services
  12. 12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. A simple mantra Prefer Services to Software Make services that are robust and functional. Buy services that other folk do better than we can for the time and/or money. Prefer Software to People Automate everything where possible. Output actionable telemetry for all the things. Prefer People to Bureaucracy Trust in the people you've employed to do the right thing and do it well. Remove unnecessary paperwork and processes whenever you can. Prefer ChatOps for Everything Email is so 1990's. Put everything on Slack so everyone can see it and action
  13. 13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. AWS at Hive – Because Reasons Suitable Fits with our Prefer Services to Software mindset Ubiquitous Easy to source talent who are familiar with the service Reliable Good support, good uptime, can be engineered for failure resilience Adaptable Not just servers & databases
  14. 14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Hive & AWS - Growing together
  15. 15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Company growth 2012 - Startup Lean Enterprise under British Gas 2014 - Larger business Product & customer base growth. Scaling & expansion 2016 - Partner Acquisition Acquired our hardware and platform partner. Merged the teams and functions 2017/2018 - International growth Launched in Ireland, US, Canada, Italy
  16. 16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. AWS account growth 2 15 32 89 0 10 20 30 40 50 60 70 80 90 100 2012 - Startup - Dev & Prod 2014 - Larger Business - Start of multi account strategy 2016 - Partner Acquisition - Merged many new accounts 2017/2018 - International growth & multi account optimisation AWS Accounts
  17. 17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. AWS at Hive - Now One AWS Account per product and/or environment Currently 110 accounts and growing (active and legacy) Is that big? No, but it’s not small either Large data volume Over 100,000 points per second of operational telemetry alone Over 230,000 log files per day from AWS CloudTrail Over 7 billion searchable documents in 12 Amazon Elasticsearch Service logging clusters Enterprise support Better pricing model, better support & direct contact with product teams
  18. 18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Our multi account journey
  19. 19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Why multi account? Separation of responsibility Cost attribution by product or function Reduced blast radius for changes Clear security boundary Easier account limit management - resources, API calls, I/O
  20. 20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Early challenges Manual AWS Identity & Access Management user control No consistency in account naming, user naming, account usage or resource tagging Complex cost attribution under consolidated billing Wild west for development teams - no oversight Third party contributors
  21. 21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Issues we found with multi account Amazon VPC peering - IP range clashes Keeping track - Multi region * multi account = tons of places to manage stuff AWS Identity & Access Management users * many AWS Accounts = tons of unmanaged users Logging everything - how to parse & where to store Many accounts, no consistency due to growth speed
  22. 22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Multi Account standardisation
  23. 23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Multi account standardisation Consistent account naming product_or_function-environment-geographic_location Consistent notifications Standardised email addresses for all teams & root accounts Consolidated notification in visible places Instance events, monitoring & deployments in Slack Consistent Security Root user 2FA, AWS Cloudtrail everywhere, Amazon GuardDuty everywhere
  24. 24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Example accounts at Hive Master/payer account Empty & restricted access. Used for AWS Organizations Sensitive, restricted accounts Security admin, centralised logging, backups, operational services Product & function accounts At least one production account and one non-production account per product or function Isolated product or function accounts Stand alone accounts for proof of concept or research
  25. 25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Example accounts masterbilling centrallog-prod security-prod ops-prod internal_it-prod test_product1-poc research-poc product1-dev product1-prod product1-prod-emea product2-dev product2-stage product2-prod function1-dev function1-prod-apac
  26. 26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Multi account - AWS Organizations Using AWS Organizations enabled Automation of standardised AWS Account creation Use of Organization Units for programmatic assignment of accounts Use of Service Control Policies (SCPs) to centrally manage high-level permissions
  27. 27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Account creation automation – a rough guide $ aws --profile ${PROFILE} organizations create-account --email ${EMAIL} --account-name ${NEWACCNAME} --iam-user-access-to- billing DENY $ aws --profile ${PROFILE} organizations list-create-account-status - -states SUCCEEDED | grep ${NEWACCNAME} $ aws --profile ${PROFILE} sts assume-role --role-arn arn:aws:iam::${NEWACCNUMBER}:role/OrganizationAccountAccessRole - -role-session-name sample # Using assumed credentials in the new account $ aws --region eu-west-1 cloudformation create-stack --stack-name operational-roles --template-body file://operational-roles- cf.yaml --region eu-west-1 && cloudformation wait stack-create- complete --stack-name operational-roles $ aws iam create-account-alias --account-alias ${ALIAS} $ aws iam update-account-password-policy --minimum-password-length 20
  28. 28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Account creation manual tasks Complete the setup of the account Tax settings and support package Enable root account security Set root account password using the forgotten password process Enable root account multi-factor authentication Set Alternative Contacts For team notifications Document and communicate
  29. 29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Our standardisation tools Hive Bill-O-Matic Consolidated billing reporting and attribution tool Security Monkey By Netflix, alerting to Slack https://github.com/Netflix/security_monkey Elastatus Read only view of all resources https://github.com/mindcandy/elastatus Hive Centralised Logging Service Fully managed log aggregation service for developer teams. Uses Amazon Elasticsearch Service
  30. 30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. From standardised to optimised Image By Balu Ertl - Own work, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=38531293
  31. 31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Current progress AWS Organizations for account management AWS CloudFormation for configuration management CHAIM for AWS IAM user management Amazon GuardDuty for network security monitoring Amazon Inspector for instance level security monitoring AWS CloudFormation StackSets for account deployment consistency Wavefront for reserved & spot instance and cost optimisation Exploring AWS Systems Manager & Amazon Kinesis Analytics
  32. 32. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Configuration management AWS CloudFormation for infrastructure configuration management Puppet for instance configuration management SAM and Serverless for AWS Lambda deployment management AWS CodePipeline, AWS CodeBuild & AWS CodeDeploy for automated CI/CD AWS CloudFormation StackSets for account deployment consistency initially but now moving to AWS CodePipeline
  33. 33. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. CHAIM for IAM user management CHAIM – Centrica Hive Access and Identity Manager Self service access to centrally managed, time limited, least privilege, assumed role credentials and pre-signed console URLs Amazon Cognito federated to AD for user authentication Amazon RDS user database for user authorisation Amazon API Gateway & AWS Lambda for request processing CLI & Slack clients CLI tool automates ~/.aws/credentials file management
  34. 34. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. CHAIM architecture Slack & CLI Client Amazon Cognito federated to AD for authentication MySQL User DB for authorization Predefined, centrally managed, least privilege roles in all accounts for assumption by CHAIM
  35. 35. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon GuardDuty for security monitoring Near real time network and AWS log analysis Built in rulesets Killer feature - Monitors the AWS APIs AWS Lambda to Slack for customised alerts and notifications
  36. 36. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Amazon GuardDuty architecture Slack & Email output Amazon CloudWatch Events to channel/identify events AWS Lambda to process and alert on events
  37. 37. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Wavefront.com for account telemetry SaaS telemetry, monitoring and alerting Any time series metric can be ingested Automated ingestion from AWS CloudWatch Programmatic analysis of all AWS resources in all accounts to allow observability and optimisation of Reserved instances - in use/available Spot instances - in use Underutilised resources e.g. EC2, EBS, EIP, ELB, DynamoDB capacity, etc
  38. 38. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Wavefront.com example dashboard
  39. 39. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Security Monkey Security Monkey monitors accounts for policy changes and alerts on insecure configurations Allows for teams to record justifications for anomalous configurations that would otherwise be deemed insecure e.g. public Amazon S3 bucket for static content hosting Also functions as a record of all current and past resources in our AWS estate
  40. 40. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Security Monkey architecture Assumed roles in each monitored account for resource scanning AWS Lambda to process and alert on events and output to Slack
  41. 41. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Security Monkey and beyond Security Monkey enhancements Alerts to central channel and to AWS Account owners Amazon Kinesis streaming of data with AWS Lambda performing in-stream automated triage of events AWS Lambda performing automated remediation based on common triage outcomes Other Security work AWS Systems Manager Run Command and Agent as an SSH replacement
  42. 42. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Stumbling blocks Image from http://www.brainlesstales.com/ Used with permission
  43. 43. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Current limitations with AWS Services AWS CloudFormation StackSets Got us going quickly but hit limitations and overhead of re-deployment Workaround using AWS CodePipeline / AWS CodeBuild AWS Organizations Service Control Policies are not fine-grained enough Amazon GuardDuty No overall security dashboard / overview for multiple regions when aggregating centrally
  44. 44. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Workarounds & mitigations How we workaround the current limitations Working closely with the Amazon AWS Service Teams to feed back and improve their services Detailed, specific help via support tickets Enterprise support enables quick resolution and feedback on issues
  45. 45. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Current issues with our tools Log Volume Who has time to look at all the logs?! Future work to log via Kinesis Streams. Processed by Lambda, in flight, with automated analysis, triage and possibly remediation Buy In "You can't install your Lambdas / Cloudformation stacks / IAM roles in my account! That'll be far too confusing!" Engagement with the process as a value-add
  46. 46. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Speaker Contact SRE Security Consultant Ronin IT Consulting for Centrica Hive @varspare https://roninitconsulting.com Mark Davison SRE Technical Lead Centrica Hive @PercussiveFix https://www.percussiverepair.net Jay Harrison
  47. 47. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Please complete the session survey in the summit mobile app.
  48. 48. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Thanks!

×