Vanguard and Bloomberg's use of AWS PrivateLink as they moved from a small number of large accounts to a large number of small accounts reduced blast radius at the management plane but introduced significant complexity at the network layer. In this session, we introduce the type of network segmentation that is required to implement a zero-trust network for a highly regulated financial investment company like Vanguard—one that adds additional complexity.

2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
How Vanguard and Bloomberg
Use AWS PrivateLink
N E T 3 2 3
Ilya Epshteyn
Principal Solutions Architect
Amazon Web Services
Barry Sheward
Chief Enterprise Architect
Vanguard
Cory Albert
Global Head of Cloud Strategy
Bloomberg

3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Agenda
AWS PrivateLink overview
Vanguard’s use of AWS PrivateLink as part of micro account
strategy
Bloomberg’s use of AWS PrivateLink for real-time data (B-PIPE)

4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
VPC
Peering
The
Internet
On-Premises
VPC
VPN
AWS
Direct Connect
Availability Zone A Availability Zone B
Instance C
10.1.3.33/24
Instance A
10.1.1.11/24
Instance B
10.1.2.22/24
Instance D
10.1.4.44/24
Public Subnet Public Subnet
Private Subnet Private Subnet
NAT
VGW
IGW
EIP: 54.1.13.43=10.1.1.11
NAT Gateway
AWS network primer (prior to AWS PrivateLink)

5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
VPC endpoint gateway type
• Limited support—Amazon S3 and Amazon DynamoDB only
• Gateway endpoints not accessible from on-prem network
natively (requires somewhat complex proxy setup)
• Available only for AWS services

6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
VPC peering
• Designed for use cases with broad, bi-directional network trust
• Not intended for fine-grained microservices trust model
• Maximum of 125 peering connections per VPC by design
• VPCs cannot have overlapping CIDR blocks

7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Connection always initiated
by the service user
Brings services into your VPC
and on-premise network via
AWS private network
Service owner only exposes a
service concept without any
network complexity
AWS PrivateLink enables a secure and scalable model
for sharing services

8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
PrivateLink for AWS Services, Enterprises, and Partners
18 AWS Services (and growing)
AWS KMS Amazon
Kinesis
AWS STS Amazon
SNS
Amazon EC2
Systems Manager
Amazon
EC2 APIs
Amazon API
Gateway
Amazon
CloudWatch
AWS Direct
Connect
VPN
Connection
Your Shared Services in Another AWS Account and VPC
AWS Partners / Marketplace
corporate data center

9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Natively accessible from on-prem networks
AWS Partners / Marketplace
18 AWS Services (and growing)
AWS KMS Amazon
Kinesis
AWS STS Amazon
SNS
Amazon EC2
Systems Manager
Amazon
EC2 APIs
Amazon API
Gateway
Amazon
CloudWatch
Your Shared Services in Another AWS Account and VPC
AWS Direct
Connect
VPN
Connection
corporate data center

10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Consumer VPC
Service Provider VPC
Application/Service
NLB
PrivateLink Architecture

11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
VPC Endpoint: vpce-2222.foo.amazon.com
NLB
PrivateLink Architecture
Consumer VPC
Service Provider VPC
Application/Service

12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
10.0.0.0/16
Consumer VPC
Private 2a
10.0.16.0/20
Private 2b
10.0.32.0/20
10.0.16.1
10.0.32.2
10.0.0.0/16
Provider VPC
Public 2a
10.0.1.0/20
Destination Target
10.0.0.0/16 Local
0.0.0.0/0 IGW
Destination Target
10.0.0.0/16 Local
Private 2a
10.0.16.0/20
Private 2b
10.0.32.0/20
10.0.16.1
10.0.32.1

13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
10.0.0.0/16
Consumer VPC
Private 2a
10.0.16.0/20
Private 2b
10.0.32.0/20
10.0.16.1
10.0.32.2
10.0.0.0/16
Provider VPC
Public 2a
10.0.1.0/20
Private 2a
10.0.16.0/20
Private 2b
10.0.32.0/20
10.0.16.1
10.0.32.1
corporate data center

14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
VPC 10.0.0.0/16
Public 2a
10.0.16.0/20
Private 2a
10.0.144.0/20
Amazon KMS
(Provider)
Destination Target
10.0.0.0/16 Local
0.0.0.0/0 IGW
Destination Target
10.0.0.0/16 Local
10.0.158.56
10.0.128.238
kms.us-east-1.amazonaws.com
vpce-042260d8dadad476a-0vjawe46.kms.us-east-1.vpce.amazonaws.com
vpce-042260d8dadad476a-0vjawe46-us-east-1a.kms.us-east-1.vpce.amazonaws.com
vpce-042260d8dadad476a-0vjawe46-us-east-1b.kms.us-east-1.vpce.amazonaws.com
Endpoint-specific DNS and
Default service DNS
Endpoint-specific DNS and
Default service DNS
with “Enable Private DNS feature” (recommended)

15. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
VPC peering and VPC endpoints
• Broad-based network trust
• Connect VPCs, not services
• Inter-region connectivity
• Fine-grained trust between services
• Service provider and consumer
• Scalable to thousands of consumers
VPC peering VPC endpoints with AWS PrivateLink

16. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Key benefits
• Private IP addresses used to connect to external services
• Same reliable and scalable technology used to access AWS services,
Enterprise microservices, or third-party solutions
• Support for overlapping addresses and reduced management points
• Service owner only exposes a service concept
• Connection always initiated by the service user
• Accessible from VPC or from on-prem (DX or VPN – NEW)
• Growing support by AWS services

17. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

18. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Vanguard—Background
Began
operations –
May 1, 1975 in
Valley Forge, PA
One of the world's largest investment
companies, offering a large selection of low-cost
mutual funds, ETFs, advice, and related services
Wall ST

19. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Vanguard’s account strategy—2016
AWS Account
VPC
AWS Account
VPC
AWS Account
VPC
AWS Account
VPC
AWS Account
VPC
AWS Account
VPC
AWS Account
VPC
AWS Account
VPC
AWS Account
VPC
AWS Account
VPC
AWS Account
VPC
AWS Account
VPC
DC1 DC2 DCx

20. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Issues with the 2016 approach
AWS Account AWS Account
Subnet
Network ACL
Availability Zone - A
Subnet
Network ACL
Availability Zone - B
Subnet Subnet
Network ACL Network ACL
CIDR: 192.168.0.1/26
Subnet
Network ACL
Availability Zone - A
Subnet
Network ACL
Availability Zone - B
Subnet Subnet
Network ACL Network ACL
CIDR: 192.168.1.0/24
CIDR: 192.168.0.0/28 CIDR: 192.168.0.48/28
CIDR: 192.168.0.32/28 CIDR: 192.168.0.16/28
CIDR: 192.168.1.0/26 CIDR: 192.168.1.64/26
CIDR: 192.168.1.128/28 CIDR: 192.168.191/28

21. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Key enablers for Vanguard’s micro accounts
AWS
CloudFormation
StackSets
AWS
Organizations Amazon EC2
systems
manager

22. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Vanguard’s micro account strategy
AWS Organizational
Unit
AWS
Account
AWS Organizational
Unit
AWS Organizational
Unit
AWS Organizational
Unit
AWS Organizational
Unit
AWS Organizational
Unit
AWS Organizational
Unit
AWS Organizational
Unit
AWS Organizational Unit
AWS Organizational Unit
AWS Organizational Unit
AWS Organizational Unit
AWS Organizational Unit
AWS Organizational Unit
AWS Organizational Unit
AWS Organizational Unit
AWS Organizational Unit
AWS Account
AWS Organizational Unit
AWS Account AWS Account
AWS Organizational Unit
AWS Account
AWS Organizational Unit
AWS Account AWS Account
AWS Organizational
Unit
AWS
Account
AWS Organizational Unit
AWS
Account
AWS
Account
AWS
Account
AWS
Account
AWS
Account
AWS
Account
AWS
Account
AWS
Account
AWS
Account
syslevel division account type

23. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
IAM in micro accounts—STS IdP User memberOf Description
Inan RootOU IAM Admin
Bob DevLOB#1OU LOB DevOps
Alice ProdOU Prod Support
IAM for
Enterprises:
How Vanguard Has
Matured Their IAM
Controls to Support a
Micro Account Strategy

24. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Vanguard cloud registry service
DCx
AWS Account
Transit Account
AWS Account
AWS Organizational Unit
AWS Account -
SvcConsumer
Subnet
Availability Zone - A
Subnet
Availability Zone - B
CIDR: 172.31.0.0/16
Endpoints
Endpoints
Endpoint Service
Endpoint
VCRS Endpoint Service
VCRS Endpoint
VCRS Endpoint
VCRS Endpoint
Endpoint Service #2
Endpoint Service #1
Endpoint Service #3
AWS Account - SvcProvider
Subnet
Availability Zone - A
Subnet
Availability Zone - B
CIDR: 172.31.0.0/16

25. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Vanguard cloud registry service—Building AWS PrivateLinks
AWS Account – Svc Provider
CIDR: 172.31.0.0/16
Subnet – AZ A
Subnet – AZ B
Subnet – AZ C
Subnet – AZ D
AWS Account – Svc Consumer
CIDR: 172.31.0.0/16
Subnet – AZ A
Subnet – AZ B
Subnet – AZ C
Subnet – AZ D
SvcProvider SvcConsumer
2. Endpoint Creation
SvcConsumer

26. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Access to micro accounts
AWS Account
rolerole
AWS Account
role
role
AWS Account
Bastion Account
rolerole
AWS Account
Subnet
Availability Zone - A
Subnet
Availability Zone - B
CIDR: 172.31.0.0/16
Account
role
Amazon EC2
systems manager

27. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Future vision
Ephemeral accounts
Three Rs of enterprise security1
• Rotate
• Repave
• Repair
applied to AWS Accounts
Supports ZeroAccess
Fit-for-purpose accounts
Handle special cases, for
example, custom address
ranges, VPC peering
Standard build mechanism
(VCRS)
Both Use AWS CloudFormation
post-account creation
AWS Account
1 https://builttoadapt.io/the-three-r-s-of-enterprise-security-rotate-repave-and-repair-f64f6d6ba29d

28. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.

29. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Bloomberg B-PIPE at a glance: The solution
 What is B-PIPE?
Consolidation, distribution, and access via a common API
Bloomberg Customers
330 Exchanges
5,000 Pricing Contributors
35 Million Instruments
110 Countries
80 Billion ticks/day
15k Customer Locations
2+ Servers per Location
2+ Routers per Location
Fault Tolerant Connectivity

30. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Bloomberg B-PIPE at a glance: Our customers
• Who leverages B-PIPE data?
• Capital markets professionals: Small hedge funds to international banks
• Front office applications used to
• Assess risk
• Manage portfolios
• Make informed decisions
• What drives customer buying decisions?
• Total cost of ownership
• Trust: managed service w/highly-specialized support
• Optimized: Reliability, scalability, flexibility
• These same “ilities” are driving their cloud migration . . .

31. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Our customer’s path to optimized
Pre-2017: Customers
subscribed to data on-prem
and published it to applications
residing in the cloud
Jun 2017: Zero footprint offerings
provide data directly to
applications. However, a reliance on
the internet causes performance,
reliability, and scalability concerns
Nov 2018: B-PIPE in AWS is
introduced as a cloud-optimized
solution
PublicCloud
ApplicationsTickerplants
API Infrastructure
BLOOMBERG
Parsers
Customer Prem
B-PIPE B-PIPE
App App App App
3rd
Party Content
PublicCloud
Applications
Customer Prem
Tickerplants
Distribution
BLOOMBERG
Parsers
blp
api
3RD
Party Content
Tickerplants
Distribution
BLOOMBERG
Parsers
PublicCloud
Customer VPC
Customer A Office
Bloomberg VPC
Apps
blp
api
BPIPE
3RD
Party Content

32. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Requirements for an optimized solution
• Must be a “no compromise” offering
• Content: depth and breadth
• Volumes of data consumed
• Resiliency
• Latency
• Must continue to be a managed solution
• Monitoring the health of the data path
• SW upgrades
• Entitlements management
• API consistency (BLPAPI) whether cloud, on-prem, etc.

33. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
B-PIPE optimized: Getting to the cloud (US East)
Bloomberg Global Network
Content
Providers
Content
Providers
Content
Providers
BloombergVPC
Cust A VPC
B-PIPE Cust B
B-PIPE Cust A
B-PIPE Cust B
BLPAPI
Cust A App
BLPAPI
Cust B App
B-PIPE Cust A
• Bloomberg ingests,
normalizes, and distributes
data globally
• Distribution extends to AWS
US East 1
• B-PIPE endpoints are deployed
on EC2 instances in a
Bloomberg managed VPC
• Customer applications remain
in their own VPC
• Applications connect to B-PIPE
using AWS PrivateLink
• Result: Customers no longer
need to host infrastructure to
obtain reliable market data
Cust B VPC

34. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
B-PIPE optimized: Inside the cloud (US East)
Deployment for a Single Customer Location
• 10 gig Direct Connects
• Bloomberg AFN’s optimize BW
utilization
• B-PIPE service runs on EC2
instances
• AZ’s provide resiliency
• Bloomberg provisions B-PIPE
via NLB
• Provisioned customers create
VPC endpoints to the NLB
• Optional customer private DNS
using Route 53

35. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Why we selected AWS PrivateLink? Customer Facing Monitoring Tools
Bloomberg Operations Teams Tools

36. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Results from the lab
Scenario Results
%-tiles (September 24-8)*
 DIFF 50% tile = ~0 ms
 DIFF 99% tile = ~0 ms
*
* Ranges selected solely due to AWS presentation due dates.

37. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Up Next
• Business
• Continue to work with early adopters
• Prepare for general US release
• Expand offering to meet customer demand globally
• Technology
• Develop and test multi-tenant solutions
• Auto Scale w/Load Balancing
• Expand the use of serverless

38. Thank you!
© 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.
Barry Sheward, Chief Enterprise Architect,
barry_p_sheward@vanguard.com
Cory Albert, Head of Cloud Strategy, Bloomberg Enterprise Data
calbert3@bloomberg.net

39. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved.