Multi-Account Strategy and Security with Centrica Hive

1. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Jay Harrison
SRE Technical Lead - Centrica Hive
Mark Davison
SRE Security Consultant - Ronin IT Consulting for Centrica Hive
AWS Multi Account
Management & Security
A new scaling challenge

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
A brief history of Hive

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
2011 - British Gas Remote Heating Control
Before Hive, British Gas developed RHC
in partnership with AlertMe Ltd
Ahead of its time but limited demand
No significant device design
improvement from older non-smart
thermostats
Moderately successful - 100k
customers, mostly via British Gas
upsell

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
2012
We started as British Gas Connected Homes
12 people in a borrowed basement office
Building on lessons learned from the
British Gas Remote Heating Control
product

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
2013
First IoT product released
Hive Active Heating v1
V1 Thermostat

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
2015
Hive Active Heating v2
Designed in conjunction with Yves Behar
New features - improved UI and holiday mode
V2 Thermostat

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
2016
Smart plugs, sensors, smart bulbs
Hive Camera
First diagnostic product - Boiler IQ
Smart devices Hive Camera

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
2017
Second diagnostic product - Hive Leak Detector
3rd party services
Amazon Echo, IFTTT, Google Home, Philips Hue
Hive Leak Boiler IQ

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
2018
Hive View Camera
Better features including event detection and event
history
Hive View

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
SRE at Centrica Hive

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
SRE at Hive
We run the systems, services and tools used by our
product development teams
Service Engineers
Building & maintaining development and product tools
and services
Product Engineers
Using and refining the services. Embedded in the
product teams
Security Engineers
Writing tools to secure the infrastructure, users and
services

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
A simple mantra
Prefer Services to Software
Make services that are robust and functional.
Buy services that other folk do better than we can for the time and/or
money.
Prefer Software to People
Automate everything where possible.
Output actionable telemetry for all the things.
Prefer People to Bureaucracy
Trust in the people you've employed to do the right thing and do it well.
Remove unnecessary paperwork and processes whenever you can.
Prefer ChatOps for Everything
Email is so 1990's. Put everything on Slack so everyone can see it and action

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS at Hive – Because Reasons
Suitable
Fits with our Prefer Services to Software mindset
Ubiquitous
Easy to source talent who are familiar with the service
Reliable
Good support, good uptime, can be engineered for
failure resilience
Adaptable
Not just servers & databases

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Hive & AWS - Growing together

15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Company growth
2012 - Startup
Lean Enterprise under British Gas
2014 - Larger business
Product & customer base growth. Scaling & expansion
2016 - Partner Acquisition
Acquired our hardware and platform partner. Merged the
teams and functions
2017/2018 - International growth
Launched in Ireland, US, Canada, Italy

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS account growth
2
15
32
89
0
10
20
30
40
50
60
70
80
90
100
2012 - Startup - Dev & Prod 2014 - Larger Business -
Start of multi account
strategy
2016 - Partner Acquisition -
Merged many new accounts
2017/2018 - International
growth & multi account
optimisation
AWS Accounts

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
AWS at Hive - Now
One AWS Account per product and/or environment
Currently 110 accounts and growing (active and legacy)
Is that big? No, but it’s not small either
Large data volume
Over 100,000 points per second of operational telemetry alone
Over 230,000 log files per day from AWS CloudTrail
Over 7 billion searchable documents in 12 Amazon Elasticsearch
Service logging clusters
Enterprise support
Better pricing model, better support & direct contact with product
teams

18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Our multi account journey

19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Why multi account?
Separation of responsibility
Cost attribution by product or function
Reduced blast radius for changes
Clear security boundary
Easier account limit management - resources, API calls, I/O

20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Early challenges
Manual AWS Identity & Access Management user
control
No consistency in account naming, user naming,
account usage or resource tagging
Complex cost attribution under consolidated billing
Wild west for development teams - no oversight
Third party contributors

21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Issues we found with multi account
Amazon VPC peering - IP range clashes
Keeping track - Multi region * multi account = tons of
places to manage stuff
AWS Identity & Access Management users * many AWS
Accounts = tons of unmanaged users
Logging everything - how to parse & where to store
Many accounts, no consistency due to growth speed

22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Multi Account standardisation

23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Multi account standardisation
Consistent account naming
product_or_function-environment-geographic_location
Consistent notifications
Standardised email addresses for all teams & root accounts
Consolidated notification in visible places
Instance events, monitoring & deployments in Slack
Consistent Security
Root user 2FA, AWS Cloudtrail everywhere, Amazon
GuardDuty everywhere

24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Example accounts at Hive
Master/payer account
Empty & restricted access. Used for AWS Organizations
Sensitive, restricted accounts
Security admin, centralised logging, backups, operational
services
Product & function accounts
At least one production account and one non-production
account per product or function
Isolated product or function accounts
Stand alone accounts for proof of concept or research

25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Example accounts
masterbilling
centrallog-prod
security-prod
ops-prod
internal_it-prod
test_product1-poc
research-poc
product1-dev
product1-prod
product1-prod-emea
product2-dev
product2-stage
product2-prod
function1-dev
function1-prod-apac

26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Multi account - AWS Organizations
Using AWS Organizations enabled
Automation of standardised AWS Account
creation
Use of Organization Units for programmatic assignment
of accounts
Use of Service Control Policies (SCPs) to centrally manage
high-level permissions

27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Account creation automation – a rough guide
$ aws --profile ${PROFILE} organizations create-account --email
${EMAIL} --account-name ${NEWACCNAME} --iam-user-access-to-
billing DENY
$ aws --profile ${PROFILE} organizations list-create-account-status -
-states SUCCEEDED | grep ${NEWACCNAME}
$ aws --profile ${PROFILE} sts assume-role --role-arn
arn:aws:iam::${NEWACCNUMBER}:role/OrganizationAccountAccessRole -
-role-session-name sample
# Using assumed credentials in the new account
$ aws --region eu-west-1 cloudformation create-stack --stack-name
operational-roles --template-body file://operational-roles-
cf.yaml --region eu-west-1 && cloudformation wait stack-create-
complete --stack-name operational-roles
$ aws iam create-account-alias --account-alias ${ALIAS}
$ aws iam update-account-password-policy --minimum-password-length 20

28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Account creation manual tasks
Complete the setup of the account
Tax settings and support package
Enable root account security
Set root account password using the forgotten password
process
Enable root account multi-factor authentication
Set Alternative Contacts
For team notifications
Document and communicate

29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Our standardisation tools
Hive Bill-O-Matic
Consolidated billing reporting and attribution tool
Security Monkey
By Netflix, alerting to Slack
https://github.com/Netflix/security_monkey
Elastatus
Read only view of all resources
https://github.com/mindcandy/elastatus
Hive Centralised Logging Service
Fully managed log aggregation service for developer teams. Uses
Amazon Elasticsearch Service

30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
From standardised to optimised
Image By Balu Ertl - Own work, CC BY-SA 4.0, https://commons.wikimedia.org/w/index.php?curid=38531293

31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Current progress
AWS Organizations for account management
AWS CloudFormation for configuration management
CHAIM for AWS IAM user management
Amazon GuardDuty for network security monitoring
Amazon Inspector for instance level security monitoring
AWS CloudFormation StackSets for account deployment
consistency
Wavefront for reserved & spot instance and cost optimisation
Exploring AWS Systems Manager & Amazon Kinesis
Analytics

32. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Configuration management
AWS CloudFormation for infrastructure configuration
management
Puppet for instance configuration management
SAM and Serverless for AWS Lambda deployment
management
AWS CodePipeline, AWS CodeBuild & AWS CodeDeploy for
automated CI/CD
AWS CloudFormation StackSets for account deployment
consistency initially but now moving to AWS
CodePipeline

33. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
CHAIM for IAM user management
CHAIM – Centrica Hive Access and Identity Manager
Self service access to centrally managed, time limited, least
privilege, assumed role credentials and pre-signed
console URLs
Amazon Cognito federated to AD for user authentication
Amazon RDS user database for user authorisation
Amazon API Gateway & AWS Lambda for request
processing
CLI & Slack clients
CLI tool automates ~/.aws/credentials file management

34. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
CHAIM architecture
Slack & CLI Client
Amazon Cognito
federated to AD for
authentication
MySQL User DB for
authorization
Predefined,
centrally managed,
least privilege roles
in all accounts for
assumption by
CHAIM

35. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon GuardDuty for security monitoring
Near real time network and AWS log analysis
Built in rulesets
Killer feature - Monitors the AWS APIs
AWS Lambda to Slack for customised alerts and
notifications

36. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Amazon GuardDuty architecture
Slack & Email
output
Amazon
CloudWatch
Events to
channel/identify
events
AWS Lambda to
process and alert
on events

37. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Wavefront.com for account telemetry
SaaS telemetry, monitoring and alerting
Any time series metric can be ingested
Automated ingestion from AWS CloudWatch
Programmatic analysis of all AWS resources in all
accounts to allow observability and optimisation of
Reserved instances - in use/available
Spot instances - in use
Underutilised resources e.g. EC2, EBS, EIP, ELB,
DynamoDB capacity, etc

38. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Wavefront.com example dashboard

39. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Security Monkey
Security Monkey monitors accounts for policy
changes and alerts on insecure configurations
Allows for teams to record justifications for
anomalous configurations that would otherwise be
deemed insecure e.g. public Amazon S3 bucket for
static content hosting
Also functions as a record of all current and past
resources in our AWS estate

40. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Security Monkey architecture
Assumed roles in
each monitored
account for
resource scanning
AWS Lambda to
process and alert
on events and
output to Slack

41. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Security Monkey and beyond
Security Monkey enhancements
Alerts to central channel and to AWS Account owners
Amazon Kinesis streaming of data with AWS Lambda
performing in-stream automated triage of events
AWS Lambda performing automated remediation
based on common triage outcomes
Other Security work
AWS Systems Manager Run Command and Agent as
an SSH replacement

42. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Stumbling blocks
Image from http://www.brainlesstales.com/ Used with permission

43. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Current limitations with AWS Services
AWS CloudFormation StackSets
Got us going quickly but hit limitations and overhead
of re-deployment
Workaround using AWS CodePipeline / AWS
CodeBuild
AWS Organizations
Service Control Policies are not fine-grained enough
Amazon GuardDuty
No overall security dashboard / overview for multiple
regions when aggregating centrally

44. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Workarounds & mitigations
How we workaround the current limitations
Working closely with the Amazon AWS Service Teams
to feed back and improve their services
Detailed, specific help via support tickets
Enterprise support enables quick resolution and
feedback on issues

45. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Current issues with our tools
Log Volume
Who has time to look at all the logs?!
Future work to log via Kinesis Streams.
Processed by Lambda, in flight, with
automated analysis, triage and possibly
remediation
Buy In
"You can't install your Lambdas /
Cloudformation stacks / IAM roles in my
account! That'll be far too confusing!"
Engagement with the process as a value-add

46. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Speaker Contact
SRE Security Consultant
Ronin IT Consulting for Centrica
Hive
@varspare
https://roninitconsulting.com
Mark Davison
SRE Technical Lead
Centrica Hive
@PercussiveFix
https://www.percussiverepair.net
Jay Harrison

47. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Please complete the session survey
in the summit mobile app.

48. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Thanks!