Vietnam AWS Community Day 2018

Secure video content with
AWS Elemental Media
Services

Profile
Name: Kien Nguyen
Country: Vietnam
Work: Cloud Solution Architect
Company: OSAM
UserGroup: AWSVN-UG

EXPLOSION OF CONTENT AND
DEVICES❏ Age of content abundance: everyone is creating video
❏ Devices everywhere: at end of 2017, an estimated 1.6 billion people worldwide watch
online video on connected devices
❏ OTT services: at end of 2016, more than 49 million homes accessed at least one OTT
service
❏ Video quality 4K and HDR

BUT
,Today’s infrastructure is still too complex
Months and Years to
deploy

Broadcast quality workflows in minutes

Challenges of On-prems
TranscodingComplex to setup and manage
infrastructure
Scaling is challenging Large investments up-front

AWS Elemental
MediaConvert
AWS Elemental MediaConvert is a file-based video processing service
that allows anyone, with any size content library, to easily and reliably
transcode on- demand content for broadcast and multi-screen delivery
❏ Access to professional grade video features and quality
❏ No software or hardware infrastructure to manage
❏ Automatically scales in response to variations in incoming video volume
❏ Ability to manage capacity and control order in which jobs are processed

AWS Elemental Media Convert
Concepts❏ Job
❏ Primary unit of work, specifies input and output
❏ Output Preset
❏ Settings to create a single output
❏ Job Template
❏ Collection of commonly used job settings
❏ Useful when processing a collection of inputs to produce a fixed set of outputs
❏ Queue
❏ All jobs are submitted to a queue
❏ Allows user to separate or group jobs for processing
❏ Jobs within a queue are processed in parallel, and queues are processed in parallel

Benefits of AWS
MediaConvert❏ Broadcast grade workflows in the cloud
❏ Shorten time to market and iterate
❏ Reduced financial risk
❏ Massively scalable and performant
❏ Robust and resilient workflow

AWS Elemental
MediaLiveAWS Elemental MediaLive is a live video processing service that
enables anyone to encode high-quality live video streams for broadcast
television and multiscreen devices
❏ Deploy live channels in minutes with resources that scale elastically based on load and
number of channel
❏ Resiliency and channel monitoring are part of the channel based pay-as-you-go service,
simplifying operational complexity and improving cost efficiency
❏ Build flexible 24x7 live video workflows or deliver event-based live streams with full
control over encoding parameters using a best-of- breed service model with support for
standard video players and a broad range of CDNs

AWS Elemental MediaLive
Workflow
❏ Broadcast-grade live encoding
❏ Easily deliver highly reliable 24x7 linear channels and live events at scale
❏ Streamlined setup, configuration, integration and operation
❏ Single control plane automatically manages resources with no need to configure redundancy
❏ Channel based pricing
❏ Predictable cost based on usage and channel parameters § No need to size or manage infrastructure or instances

Benefits of AWS Elemental
MediaLive❏ Simplify provisioning and management of live channels
❏ Reduce cost and operational overhead
❏ Provide resilient live streaming without additional complexity
❏ Maintain flexibility and control over workflow solution
❏ Utilize advanced broadcast features to build best-of-breed

AWS Elemental
MediaPackageAWS Elemental MediaPackage is a video origination and just-in-time
packaging service that allows anyone to securely and reliably deliver
streaming content at scale
❏ Offers just-in-time packaging for cost-effective video distribution using multiple delivery
and content protection standards
❏ Makes it easy to enrich audience experiences with time-shifted TV and other advanced
features
❏ Reduces workflow complexity, increases origin resiliency, and provides protection for
multiscreen content without the risk of under or over-provisioning infrastructure

Benefits of MediaPackage Live
workflow
❏ Just-in-time packaging
❏ Enables efficient distribution of multiple delivery protocols and content protection/DRM standards
❏ Support for a range of OTT devices (mobiles, tablets, desktops, connected TVs, games consoles & set-top boxes)
❏ Live-to-VOD
❏ Easily add live-to-VOD functionality for catch-up TV and start-over TV features
❏ Highly available and reliable
❏ Multi Availability Zone redundancy within a region built in and elastic scale and healing without user intervention

Benefits of AWS
MediaPackage❏ Efficient delivery video to a § broad range of devices
❏ Rich content protection features
❏ Simply enrich the audience
❏ High availability and reliability without over-investment
❏ Create a best-of-breed solution

AWS Elemental
MediaStoreAWS Elemental MediaStore gives customers the high performance
and immediate consistency required for content origination for live
and on-demand media combined with the security and durability of
Amazon Simple Storage Service (S3)
❏ It provides a cost effective method for simple pass- through content delivery with
predictable pay-as-you- go pricing
❏ The service also provides a hierarchical view of media content enabling simpler
integration with third-party systems

AWS Elemental
MediaTailorAWS Elemental MediaTailor is a content personalization and
monetization service that allows customers to implement stitched
server-side ad insertion while maintaining high quality of service
❏ Ads are better monetized, more consistent in video quality and easier to manage
across multi-platform environments
❏ Managed transcoding provides a better user experience
❏ Standards-based client and server-side ad reporting within a single service
❏ More control over the player, origin and CDN while providing a better quality of
service for end-viewers at scale

UNPROTECTED
VIDEOSYou can:
❏ Copy it
❏ Send copies to everyone you know
❏ Remix it
❏ Play on any machine

Digital Rights
Management❏ Copy it
❏ Send copies to everyone you know
❏ Remix it
❏ Play it on any a few machines

How to implement DRM with
AWS Elemental Media
Services?

Introducing
SPEKESecure Packager and Encoder Key Exchange (SPEKE) is part of the AWS Elemental content encryption protection
strategy for media services customers. SPEKE defines the standard for communication between AWS media services and
digital rights management (DRM) system key servers. SPEKE is used to encrypt video on demand (VOD) content through
AWS Elemental MediaConvert and for live content through AWS Elemental MediaPackage

How AWS saved my first data lake journey
Hyun Joong Kim
− A story of a college student in the Korean startup scene −
Presented by

ABOUT ME
Hyun Joong Kim
3
5
- Member of the KRUG/ AUSG
- Senior at Hanyang University
- Department of Information System
- Former Intern at MyMusicTaste
- Data team
- Former Intern at Ebay Korea
- Data Platform team

AGENDA
About AUSG1.
2.
3.
MyMusicTaste
Data Lake
3
6
4. Retro
5. Closing

About AUSG
AWSKrug University Student Group
1

Motivation
57
•In-development: lighter, modular, less barriers
• System-wide dependencies —> Per Service dependencies
• Decoupled parallel development/deployment
• More cool stuff!
•In-production: lighter, scalable, fault tolerant
• Single Point of Failure —> Distributed fault tolerance
• Whole application auto-scaling —> scale individual services as needed

Motivation
58
• Diversified data sources
• ElasticSearch
• DynamoDB
• PostgreSQL
• S3
• Redis
• …
..and more!

Use Cases & Characteristics
59
•Must be able to ingest and store all types of data
• Internal: transactional data, application logs, operational data
• External: programmatically and manually extracted meta data
•[Close-to-]real-time representation (for platform)
•Must be mapped to [loose]schema and queryable
•Must be source for further ETL and analytics operations
•Must have full production: secure, testing, logging, etc

Data Lake @ MMT
3
Overall Architecture

Data Lake Query Layer
61
Amazon S3 (storage for streams, snapshots, raw, pre- &
post-transformed, and final query layer data)
Amazon Glue Catalog (Hive metastore)
Amazon Athena (Presto SQL queries)
Periscope Data (dashboards, charts, visualizations)

Retro: what went well
65
•Glue catalog & Hive QL
•Automated schema discovery with Glue crawlers
•Read & write to S3 + Parquet

Retro: What could be improved
66
•Glue :(
• Bookmarks are black-boxed and demonstrate some non-deterministic issues
• Development and maintenance of Glue scripts is clunky
• Cost!!! …minimal monthly cost for running one job every 30mins:
•Streaming from Aurora PostgreSQL
2 DPUs 5 DPUs 10 DPUs
$211.20048 $528.00048 $1055.99952

Future Work: Spark + EMR
67
•Control the scaling & lifecycle of our EMR resources,
and reduce cost drastically WHILE increasing load
•Faster development
•Better management options

Future Work: E[CWL]K Framework
68

Future Work: Warehousing & Analytics
69

SERVICE
ISSUES
71
- Exposure to services for college students
- Seminars, hands-on-labs that helped
- Expose of architecture and services
- Not a 100% fit but did help know where the pieces belonged to
- Slack page for KRUG
- Community members that are interested in such projects
- Active QnA that helped in situations

Personal thoughts
72
- Would love to have more members join
- Make a bigger pool of enthusiastic students
- AWS is great, but there is no magic
- A bigger network of people who are interested in technology in
general
- Study the core of how things inside AWS work the way they do

Reference
73
1. https://datafloq.com/read/what-is-a-data-lake-what-are-the-benefits/2589
2. https://aws.amazon.com/big-data/datalakes-and-analytics/what-is-a-data-lake/
3. https://docs.aws.amazon.com/aws-technical-content/latest/building-data-lakes/amazon-s3-data-lake-storage-
platform.html
4. https://aws.amazon.com/glue/
5. http://calculator.s3.amazonaws.com/index.html

www.mymusictaste.com
Special Thanks To Paul
Elliot,
data lead at MyMusicTaste

Simplified Networking
For Multi-Region Approach

- Lead Devops Engineer, Onerent Inc.
- Ansible Lover
- Debian User
- Docker Contributor
- Blogger (cebuserver.com)
- AWSUGPH-Davao Leader

- Purchase Servers
- Purchase IP Ranges
- Lockin Contract with
Data Centers
- Configure DNS Servers
- Maintenance Cost

- VPC
- Security Groups
- User Access
Management
- High Availability
- Rugged Devops
Approach

BEST PRACTICES:
- Apply FQDN to all instances
- mydomain.com
- mydomain.net

BEST PRACTICES:
- Apply proper naming convention to all instances
- webapp-beta-us.mydomain.net
- backend-staging-sg.mydomain.net
- frontend-prod-01.us-west-1.mydomain.net
- frontend-prod-02.us-west-2.mydomain.net
- workers-aux-server.ap-southeast-1.mydomain.net

BEST PRACTICES:
- Apply ssh configuration ~/.ssh/config
HOST jumphost-sg-01
User firstname.lastname
Hostname jumphost-sg-01.mydomain.net
HOST frontend-prod-01
User firstname.lastname
Hostname frontend-prod-01.us-west-1.mydomain.net
Port 22
ProxyCommand ssh -q -W %h:%p jumphost-sg-01

BEST PRACTICES:
- Create a service user (server.deployer)
- Runs all the services required by application
- Always use processes management

BEST PRACTICES:
- Only use the root PEM to provision the users
- Apply VPN Keys Rotation (Quarterly)
- Require Office Network on Static IP Address
- Apply Log Management
- AWS CLOUDTRAIL
- AWS VPC FLOW LOG
- AWS INSPECTOR

If you cannot even secure one
place, why go for multiple entry
points setup?

Architecting around
Multiple AWS Accounts
Things you really need to know

Self Introduction
• Steve Teo
• Dev -> Build & Release -> Cloud DevOps
• 7 years working in various engineering teams
• 3 years working on AWS
• https://www.linkedin.com/in/steve-teo-b7988541/

Community
https://www.meetup.com/AWS-
SG/
https://www.meetup.com/
Atlassian-User-Group-Singapore/

Question
•How many of you are responsible for your
company’s AWS Account(s)?
•How many accounts does your company /
team have?

Purpose
For cloud architects & engineers to be
aware of the benefits and complexities of
an AWS Multi-Account Architecture

Goals
•Understand the various motivations for wanting to
separate AWS Accounts
•Understand the immediate and non-immediate
decisions you will be likely to face
•Understand gotchas, best practices

Background
?
?
?
?
?
?
?
?
?
?
?
?
?
?10.0.0.0/16 10.0.0.0/16

Background
• Previous Company
• 2 Legacy Accounts with VPCs with overlapping CIDRs
• Non-Production and Production workloads sitting in the same Account
YIKES!
• Eventually migrated department workloads to 40+ AWS Accounts
• March 2017: https://speakerdeck.com/stevepotayteo/a-multi-aws-
account-story
• Current Company
• Worked on AWS Account & VPC Strategy based on enterprise
requirements
• Thinking of how to scale AWS Multi-Accounts and VPCs is one of
my weird hobbies

Lack of AWS Account or VPC Strategy
-> Massive Technical Debt
Annoying Unusable

What really needs to be done?
• Define an AWS Account Strategy
•Which should be deliberate, tailored to your
organization’s current needs and allowed to evolve
to future needs
•Removes doubt on where to place workloads

Motivations for Separating
AWS Accounts

Recap – AWS Account
What is an AWS Account?
1. Financial Responsibility
1. Billing and Financial
2. Reserved Instances
2. Resource Containment
1. Resources Boundary
2. Limits
3. Security Boundary
1. AWS User Access Security
2. Data
Ref: https://www.slideshare.net/AmazonWebServices/arc325managing-multiple-aws-accounts-at-scale - Slide 14

Recap – VPC
What is a VPC?
1. A virtual private cloud (VPC) is a
virtual network dedicated to your
AWS account. It is logically isolated
from other virtual networks in the
AWS Cloud.
2. VPCs = network containment
!= AWS User Access Security!

Separate by Business / Dev Team
• "Any organization that designs a system (defined broadly) will
produce a design whose structure is a copy of the organization's
communication structure.” – Melvin Conway
• Eg. Business – Profit Center
• Eg. Dev Team – Cost Center
• Usually done because of a need for showback / chargeback or
isolation among workloads
• Easily broken when prone to organization changes

Separate by Platform / Service /
System / Application
•Wide-grained – Platform / Service
•Fine-grained – System / Application
•Splitting it too fine-grained might not make sense at
all if it makes workloads too small
• Eg. 1 AWS account just for 1 EC2?
• Container optimization?

Separate by Environment
•By default you get
•network / data containment
•user access security
•Orthogonal to other ways of separation
•Eg. Sandbox / Non-Prod / Prod / DR
•Eg. DEV / SIT / QA / STG / PROD / DR

Other Ways
•Service Tiering (eg. Tier 1, Tier 2 services)
•PCI / HIPAA (Regulated vs Non-regulated)
•AWS Service Limits / API Rate Limits
•Limit visibility of workloads

Special Accounts
• Shared / Management Services (eg. Tools, DNS, AD)
• Landing Zone (Bastion) account
• Direct Connect (For provisioning of DX)
• Logging Account
• Security Account
• Sec Logs Account
• Transit Account (Transit VPC for hybrid connectivity)
• Backup Vault (for DR)
• Organisation Master Billing Account

Drive towards Clean Evolving
Architecture
• “The purpose of a good architecture is to defer decisions,
delay decisions.” – Uncle Bob Martin
•Decisions are driven by concerns, which can be
immediate or non-immediate depending on scale
and requirements
• Usually any concerns around networking foundation are
immediate

Recap – AWS Account & VPC
Separate AWS
accounts by definition
means separate VPCs!
You got to deal with
scaling VPCs as well!

VPC Strategy
•You need a VPC strategy
• Reference VPC Architectures
• Proper IP Address allocation to prevent
overlapping VPC CIDRs
• Maintain a proper inventory!
• VPC Peering and limitations per VPC
•Soft: 50, Hard: 125

Direct Connect Strategy
•Significantly more complex when you need to have
DX to your on-prem environment
•Pre-mature buying of DX and related services
without understanding
• Limitations of the various DX options
• AWS Account and VPC Strategy
• Interactivity between on-prem and AWS workloads
• Security and Infrastructure policies
•Engage a proper partner who can advise you

Decision Point: Transit VPC?
Ref: https://theithollow.com/2018/07/16/should-i-use-a-transit-vpc-in-aws/

Decision Point: Shared Services VPC
https://aws.amazon.com/answers/networking/aws-multiple-vpc-vpn-connection-sharing/

Centralizing Internal DNS
https://aws.amazon.com/blogs/security/how-to-centralize-dns-management-in-a-multi-account-
environment/
Authorizations that you can
create so you can associate
VPCs that were created by
one account with a hosted
zone that was created by
another account = 100

Centralizing Internal DNS
https://aws.amazon.com/blogs/security/how-to-centralize-dns-management-in-a-multi-account-
environment/

Cost Attribution
• Especially if your AWS bill goes to different cost
centers, like in an Enterprise
• You need to pre-empt and explain to your
finance team how to handle a AWS bill and do
cost attribution before they get really upset
• Better for cost transparency than cost tag
allocation

AWS Account Security
• Preventive Controls
• Use AWS Organisations – Service Control Policies to enforce global
policies, eg. Prevent disabling of Cloudtrail across all accounts
• All Root Accounts and Admin accounts to be 2FA enabled
• Apply baseline IAM policies through Cloudformation / Terraform
and CI/CD Pipelines
• Detective Controls
• Enable AWS GuardDuty

Resource Security
• Stick to established Architectural Principles and Reference
VPCs Patterns
• Build up a security model around individual resources to
understand how to secure it
• Detective & Remediative Controls
• Automated AWS Config Rules, eg. Disable global unrestricted
access to Port 22
• Enable AWS GuardDuty

Centralize Communication Channels
• Forward all emails to all the root account emails into a single
inbox to make sure no email is missed
• Incidents / Deprecations / Outages
• Setup and standardize alternate contacts

Non-Immediate Concerns
For smaller AWS Account footprints

Reduce Entropy through Infra as Code
•Are your AWS Accounts Pets or Cattle?
•Automate everything – Terraform /
CloudFormation through CI / CD pipeline
•Prevent / avoid manual changes as much as
possible to eliminate configuration draft

Logs & Alerts Management
• Infrastructure Monitoring
• Cloudwatch can’t scale beyond 1 Account. Use tools like Datadog
• Centralize Logs to Logging Accounts / Logging Infrastructure
• Cloudtrail
• Cloudwatch Logs
• Config
• S3 Logs
• ELB Logs
• Infrastructure Events
• CloudFront
• VPC Flow Logs
• Leverage on ChatOps – pump useful alerts to Slack for monitoring
discussion

Golden Image Preparation
• Need to prepare and distribute to one multiple accounts
• Use
• AWS Systems Manager Automation
• Packer
• And CI / CD Pipelines
• Document images manifest and inventory

Backup & Restore
Use proper tools like Cloud Protection Manager (CPM) to
provide single pane of glass and orchestrate across accounts
eg. DR scenario
However: Inflexible licensing around multiple AWS Accounts

Limit Unauthorized Resource
Sprawl
• “You Can’t See What You Don't Know”
• Preventive Controls
• Restrict services via AWS Organisation SCP
• Restrict regions via IAM, i.e. aws:RequestedRegion
• Detective Controls
• Enable Cloudtrail / Config

IAM User Access Management
• Insane to maintain individual IAM users on every account,
eg. 100 accounts, 10 users each
• Options
• Use a landing zone account, and STS AssumeRole to other account’s IAM
roles
• What worked well for me in the past
• AD -> Okta -> SAML Federation to every AWS Account
• Okta was good for us to manage complexity with multiple services

Gotchas
• Reserved Instances and Capacity Reservation
• RIs in a central account vs RIs in separate accounts
• Capacity Reservation only holds if specific account and specific AZs
• Resource & Charges Duplication
• Same resources provisioned across multiple accounts
• Might want to consolidate if you have the option
• Eg. Multiple NAT Gateways across AWS Accounts - Considering using Proxy
Instances
• AWS POC Credits
• Do not join a POC account to an existing Organisation, as credits can only be applied
on the master billing account! Instead let the POC account be standalone for now so
that you can apply the credits directly on it

Gotchas
• Certain services do not support single pane of glass yet
• Eg. Cloudwatch (use something like Datadog to mitigate this)
• Eg. Trusted Advisor
• Might have to roll in your own and use third party
• Some commercial tools can’t support or have punitive licenses for
Multiple AWS Accounts

Previous Gotchas (No longer Issues)
• Consolidated Billing and Reserved Instances
• Wanted selection of reserved instances specifically not against certain
accounts
• https://aws.amazon.com/about-aws/whats-new/2017/11/customize-your-
organizations-aws-credit-and-reserved-instance-ri-discount-sharing-using-
new-billing-preferences/

Tips & Best Practices
• There is no perfect design. It’s all about meeting your requirements as
best
• Evolving Architecture
• Make the most important decisions first, eg. AWS Account Structure, VPC
Segmentation
• AWS is a very complex beast. Understand the fundamentals well
• Be hands-on. Experiment and validate assumptions with hands-on
• Be aware of soft and hard limits which can impact the overall architecture,
eg. VPC Peering Limits, DX VIF
• Enforce a naming strategy, document or keep an inventory of
• AWS Accounts, VPCs

Tips & Best Practices
• Enable CloudTrail / Config by default
• Secure globally using AWS Organisations SCP where possible
• Automate AWS Account Configuration - Terraform / CloudFormation
• Centralize Everything eg. AWS Systems Manager
• Use SaaS/tools that can scale with multiple AWS Accounts (eg. Licensing,
Automation)
• Know the nature of AWS resources (eg. Account-Specific, AZ-Specific,
Region-Specific, VPC-Specific)
• Disable unused services / regions
• Do not create any workloads on the organisation master account

Learning Resources
• From One to Many: Evolving VPC
• https://www.youtube.com/watch?v=jjk_zZRLXXw 8:31
• https://www.youtube.com/watch?v=3Gv47NASmU4 20:35
• https://aws.amazon.com/answers/account-management/aws-multi-
account-security-strategy/
• https://www.slideshare.net/AmazonWebServices/arc325managing-
multiple-aws-accounts-at-scale
• http://www.glomex.com/blog/multi-account-handling/
• https://aws.amazon.com/answers/aws-landing-zone/

Tweet analysis by LDA model
Ayumi Kobukata
− I tried tweet recommendation for 3 months by a beginner at machine learning −
Presented by

ABOUT ME
Ayumi Kobukata
Shinko Technomist Co.
2013〜2017 Infrastructure engineer
2018〜 R&D of own product

OUR COMPANY
SERVICE
fusetter
2012
Service started
5,000,000
Use people
|
But when you access
the website
Some letters are hidden on Twitter Letters are revealed
https://fusetter.com/

AGENDA
Service issues1.
2.
3.
Let’s try LDA model!
Challenges for the future

SERVICE
ISSUES
98%
Page bounce rate

SERVICE
ISSUES
Page bounce rate
Tweet recommendation
98%

Machine learning
Morphological analysis
LDA modelMeCab
gensim
corpus
tf-idf
Bag of Words
Topic model

Try tweet recommendation for 3 month!
TODAY’S MAIN TOPIC
A beginner at
machine learning

WHAT IS LDA?
Amazon EC2 is a virtual server on the cloud.

WHAT IS LDA?
Amazon EC2 is a virtual server on the cloud.
talk about “AWS”

WHAT IS LDA?
LDA model
One method of topic model.
Analyze what kind of topics the document is
composed. ※Topics can not necessarily be named

PROCESS FLOW
STEP
1
STEP
2
STEP
3
STEP
4
STEP
5
STEP
6

PROCESS FLOW
STEP
1
Data collection
Collection tweet data for one day from
database.

PROCESS FLOW
Morphological analysisSTEP
2
Use “MeCab” of OSS.
Amazon / EC2 / is / a / virtual / server / on / the / cloud

PROCESS FLOW
Morphological analysisSTEP
2
Amazon / EC2 / virtual / server / cloud
Get only “nouns” and “proper nouns”
Amazon / EC2 / is / a / virtual / server / on / the / cloud

Delete URL link
TIPS
Importance of preprocessing for morphological
analysis.
あ
ア a
A
Delete one letter Unify letter of type
http://
Replacing numbers
1,2,3
0

TIPS
Importance of user dictionary
Necessary to register the name of a character in the work.

PROCESS FLOW
STEP
1
STEP
2
STEP
3
STEP
4
STEP
5
STEP
6
LDA model
Use “gensim” of OSS

PROCESS FLOW
Create a dictionarySTEP
3
Count number of word occurrences.
ID Word Count
Amazon
EC2
virtual
server
27
13
742
96
38
102
86
57
cloud6 14

PROCESS FLOW
Create a corpusSTEP
4
How many times occurrences word of the dictionary.
DocID WordID Count
1
2
3
4
1
1
1
1
1
1
1
1
WordID=1: Amazon
WordID=2: EC2
WordID=3: virtual
WordID=4: server
51 1 WordID=5: cloud

PROCESS
FLOW
Construction of LDA modelSTEP
5
Estimate at what rate included word in the topic.
topic_1: 0.048*“amazon” + 0.023*”cloud” + 0.014*”virtual”
topic_2: 0.034*“cloud” + 0.012*”google” + 0.009*”gcp”
topic_3: 0.052*“ms” + 0.037*”microsoft” + 0.026*”cloud”
：
：

ACTUAL DATA
Attack on Titan
Touken Ranbu Touken Ranbu Touken Ranbu

PROCESS FLOW
Tweet recommendationSTEP
6
Each tweet apply to which topic, and calculate
similarity to topic.
Amazon / EC2 / virtual / server / cloud
topic_1: 0.072518
：
topic_2: 0.018321
topic_3: 0.018321
1.

PROCESS FLOW
Tweet recommendationSTEP
6
Recommend a tweet with the same topic as the
tweet user read.
2.
This is the recommended tweet for you!
“Amazon VPC is virtual network service.”

TIPS
Chase of topic.
Allow the same topic in the past to be recommended.
topic_1
topic_2
topic_3
topic_4
topic_5
→ Deadpool 2
→ Golden Kamuy
→ Avengers
→ Case Closed
→ Gin Tama
：
Tweet, Title
xxxxx, Deadpool 2
xxxxx, Avengers
xxxxx, Deadpool2
：
xxxxx, Case Closed

ARCHITECTURE
Step Functions
Lambda
RunCommand
EC2
RDS
S3
Step Functions
Lambda
RunCommand
Operator

CHALLENGES FOR THE FUTURE
Migration to Amazon SageMaker
Realtime recommendation

Secure Your Landing Zone
for Enterprise
Vit Niennattrakul

Vit Niennattrakul
- Ph.D. in Data Mining (Time Series)
- AWS Community Hero
- AWS User Group – Thailand
- Managing Director @ DailiTech
- AWS External Instructor
- 8 Certifications

Securing your Infrastructure
Why is it important?

Question?
Where should I place my first resource (EC2, RDS,
etc.)?

Question?
Where should I place my first resource (EC2, RDS,
etc.)?
“Landing Zone”

Landing Zone
Default Landing Zone (Default VPC)
Singapore Region
Public Subnet
Availability Zone A
Public Subnet
Availability Zone B
Public Subnet
Availability Zone C
172.31.0.0/16
172.31.0.0/20 172.31.16.0/20 172.31.32.0/20
Internet Gateway (IGW)

Landing Zone
Default Landing Zone (Default VPC)
Singapore Region
Availability Zone A Availability Zone C
Public Subnet
Availability Zone B
Public Subnet
172.31.0.0/16
172.31.0.0/20 172.31.16.0/20 172.31.32.0/20
Internet Gateway (IGW)
EC2
Public Subnet
RDS
Public IP

Landing Zone
Do not use Default Landing Zone (Default VPC)
on Production!
Reasons
- Default subnet is public subnet.
- Default route table is attached to Internet Gateway.
- Default resources created in Default subnet will have
public IP address.

Landing Zone for Enterprise
General (Security) Requirements
- Separate permissions for developer, operations,
network, and security team on different environment
- Single sign-on to access AWS console
- Connect to on-premise
- Connect between VPCs
- Protect over DDOS attack
- Audit logs for security team
- Secure your storage

1. Create accounts to place resources
Non-production
Workload
Production
Workload
Using multi-account patterns to
separate the non-production and
production workload

2. Create IAM for each team
Development
Account
IAM
Production
Account
User A User B User C User D User A User B User C User D
IAM
Create duplicate users for
each AWS account

Development
Account
IAM
Policy for Development Team
- EC2FullAccess
Policy for Operation Team
- Administrator
Policy for Network Team
Policy for Security Team
Production
Account
IAM
- Read-only Access
- Administrator
Policy for Network Team
≠
Create IAM policy and IAM role for
each team

Development
Account
IAM
- EC2FullAccess
- Administrator
Production
Account
IAM
- Read-only Access
- Administrator
Policy for Network Team Policy for Network Team
≠
Create IAM policy and IAM role for
each team

Development
Account
IAM
- EC2FullAccess Production
Account
IAM
- Read-only Access
Policy for Operation Team Policy for Operation Team
- Administrator - Administrator
User A User B User C
Development
Team
Operation
Team
User D
Management
Account
Option 1:
Create IAM users

AWS Directory
Service AWS Managed
Microsoft AD
With AWS Managed Microsoft AD, you can easily enable your Active Directory-
aware workloads and AWS resources to use managed actual Microsoft Active
Directory in the AWS Cloud. Workload examples include Amazon EC2, Amazon
RDS for SQL Server, custom .NET applications, and AWS Enterprise IT
applications such as Amazon WorkSpaces.
AD Connector
AD Connector is a proxy for redirecting directory requests to your existing
Microsoft Active Directory without caching any information in the cloud. AD
Connector comes in two sizes, small and large. A small AD Connector is
designed for smaller organizations of up to 500 users. A large AD Connector can
support larger organizations of up to 5,000 users.

Development
Account
IAM
- EC2FullAccess Production
Account
IAM
- Read-only Access
Policy for Operation Team Policy for Operation Team
- Administrator - Administrator
Management
Account
Managed AD
Development
Team
Operation
Team
AD: Development
AD: Operation
Option 2:
Integrate with Microsoft Active
Directory

3. Connect to on-premise
Virtual Private Gateway
- Two gateways
- Passive VPN Gateway

Production
Account
Corporate Data Center
Option 1:
Create an site-to-site VPN to AWS
VPN Connection
Development
Account
VPN Connection
Customer Gateway

Production
Account
Option 2:
Using Transit Account
EC2: Network Appliance
Transit Account
Development
Account
Customer Gateway
VPN Connection
VPN Connection

Production
Account
Option 2:
Using Transit Account
Transit Account
Development
Account
Customer Gateway
Management
Account
Share
Service
Account

4. Connect between VPC
Production
Account
Transit Account
Development
Account
Customer Gateway
VPN Connection
VPN Connection
Share
Service
Account
VPC Peering
VPC Peering

5. Protect against network attack
CloudFront
- Content Delivery Network
- Support HTTP/HTTPS
- Protect against DDOS

Public Subnet
(DMZ)
Private Subnet
(Application Zone)
EC2: WorkloadApplication Load
Balancer
CloudFront

Public Subnet
(DMZ)
Private Subnet
(Application Zone)
EC2: WorkloadCloudFront NLB
(Network Load
Balancer)
IPS
(Intrusion
NLB WAF
(Network Load (Web Application
Detection System) Balancer)
Firewall)

6. Audit log for security team
Development
Account
Production
Account
Management
Account
Transit Account
S3: CloudTrail
Log Account

6. Audit log for security team
CloudTrail

7. Secure your storage
S3: Storage S3:
Encrypted
Storage
Encrypt Your S3

S3: Storage S3:
Encrypted
Storage
Encrypt Your S3 Policy for User A
- S3FullAccess
Policy for User B
- S3FullAccess
- Encrypt and decrypt keys

S3: Storage S3:
Encrypted
Storage
Encrypt Your S3 Policy for User A
- S3FullAccess
Policy for User B
- S3FullAccess
- Encrypt and decrypt keys
Cannot Access
Can Access

Suggestion
- Use CloudFormation to deploy all accounts
- Use AWS Config to keep track of configuration

Conclusion
- Separate permissions for developer, operations,
network, and security team on different environment
- Different Account
- Single sign-on to access AWS console
- Using Management Account
- Connect to on-premise
- Using Transit Account & VPN
- Connect between VPCs
- Using VPC Peering

Conclusion
- Protect network security
- CloudFront, IPS, WAF
- Audit logs for security team
- CloudTrail
- Secure your storage
- Encryption

Alexa Skill Development
for Beginner

Name: Tajiri Ayaka
Country: Japan/Osaka
Work: web application Engineer
Company: Suzukishouten
UserGroup: JAWS-UG
Profile

Virtual Assistant Developed by Amazon
Echo Dot Echo Echo Plus
Echo Spot Echo Show

Alexa Supports Several Languages
- English
- Deutsch
- French
- Japanese

You can extend Alexa's capabilities with skills
1. Custom Skill・・・Generic skills
2. Smart Home Skill・・・Skills to control home appliances and others
3. Flash Briefing Skill・・・Skills to read news etc...

Prepare
- amazon web services account
- amazon developer account
Tools
- alexa developer console
- lambda

Step5: Lambda - Create Function

Step5: Lambda - Set Name and Role

UI of the alexa developer console is wonderful !
Very easy to understand !
Please develop your skills !

• Sathyajith Bhat
• Senior DevOps Engineer - Adobe I/O
• Organizer, Bangalore AWS Users’ Group
• Author - Practical Docker with Python

Containers - How They Tend to Be

• Adobe I/O is the place for developers looking to integrate, extend, or create
apps and experiences based on Adobe's products and technologies.
• Adobe I/O API Gateway
• A performant API Gateway based on Nginx and Openresty
• 2.5 billion+ API calls per day
• Adobe I/O Events
• An event notification service to inform subscribing systems of near real-
time events happening in Adobe services.
• Adobe I/O Runtime
• A serverless platform(currently in private beta) based on Apache
OpenWhisk which allows a developer to execute code on Adobe's
infrastructure.
Adobe I/O

• From Docker Hosts
• From noisy neighbours
• From within containers
• From external world
• From within the application
Threats to/from Containers

• Control Groups (cgroups)
• Namespaces
• Kernel Capabilities
• Seccomp
• Image Security
• Vulnerability Scanning
Different mechanisms

• Group, Limit & isolate resource utilization
• Resources that can be controlled: CPU, Memory, Disk, Network
• cgroups Docker uses:
• Memory
• HugeTBL
• CPU
• CPUSet
• BlkIO
• Devices
• /sys/fs/cgroups
cgroups

• Applying limits
docker run --cpus=”0.5”
docker run --cpu-shares=512 (weighted CPU distribution, default weight ==
1024)
docker run --memory=2g
docker run --oom-kill-disable (!!)
docker run --device-read-iops
docker run --device-write-iops
• Custom cgroup?
Yes! docker run --cgroup-parent
cgroups

• Abstraction which makes a process appear they are isolated
• Controls what processes can see
• Different types of namespaces:
• Mount
• PID
• UTS
• IPC
• Network
• User
Namespaces

• Remap a user with a container to another user on the Host
• Remap privileged user within container to non-privileged one outside
host
• Enabling remapping:
dockerd --userns-remap=”remap-user:remap-group”
• Or, edit daemon.json
{
userns-remap: “remap-user”
}
Namespaces - User Namespace Remapping

• Caveats
• Ensure the users/groups are created & associated with your user
• Enable/Disable it on a new Docker install than existing one
• Can no longer user --pid=host or --network=host
Namespaces - User Namespace Remapping

• Secure Mode Computing
• Kernel feature, restricts syscalls that a process can do
• Create custom profiles, pass a different profile for each container
• Default seccomp policy for Docker
• Disables 44 system calls of 300+ system calls
seccomp

• Pre-requisites:
• Check for kernel support
• grep CONFIG_SECCOMP=/boot/config-$(uname -r)
• Apply seccomp
• docker run
• ???
• Seccomp is applied by default!
• Verify with docker info
seccomp

• Create custom profiles as json
• docker run --security-opt seccomp=profile.json
• How to find what syscalls are in place?
• strace (Linux)
• dtruss (macOS)
seccomp

cat seccomp-profile.json
{
"defaultAction": "SCMP_ACT_ALLOW",
"syscalls": [
{
"name": "chown",
"action": "SCMP_ACT_ERRNO"
},
{
"name": "chmod",
"action": "SCMP_ACT_ERRNO"
}
]
}
seccomp

/ # echo "rm -rf" > fluffy_kittens.sh
/ # chmod u+x fluffy_kittens.sh
chmod: fluffy_kittens.sh: Operation not
permitted
seccomp

• Drop unnecessary capabilities from the container
• Alternatively, provide necessary ones
• Don’t need chown capability? Drop it
• docker run --cap-drop=chown
Kernel Capabilities

• Mandatory Access Control
• Why?
• Unix permissions allow for R/W/X
• No fine grained permissions
• Why should your application look at other logs?
• Docker expects AppArmor policies to be loaded on Docker host
AppArmor

• Images are still software - and old, if not rebuilt
• Heartbleed
• Vulnerability in openSSL
• Ghost
• Vulnerability in glibc
Managing Vulnerabilities

• Vulnerability Scanners
• Twistlock
• Aqua Container Security
• Clair (CoreOS)
• Anchore
• Dagda
Managing Vulnerabilities

• Don’t use images blindly
• Host the images in private/self-hosted registry
• Publishing to Docker Hub? Enable Docker Content Trust
Trusted
Images

• Enable content trust
• export DOCKER_CONTENT_TRUST=1
• Images must have content signatures
• Trust is managed by use of signing keys
• Offline key: Root of content trust
• Repository key for signing tags
• Server managed Timestamp key
Docker Content Trust

• Kernel Capabilities
• Tutorial on Creating AppArmor Profiles
• Docker Security Docs
• Sysadmin Casts - Linux Control Groups
• Searchable Syscall Table
• Google Chrome Seccomp Sandbox Implementation Doc
• User Namespaces in Docker Engine
References

• Twitter - sathyabhat
• Email: sathya@sathyasays.com
• https://sathyasays.com
Docker Content Trust

Vietnam AWS Community Day 2018

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Vietnam AWS Community Day 2018

Similar to Vietnam AWS Community Day 2018 (20)

More from AWS Vietnam Community

More from AWS Vietnam Community (20)

Recently uploaded

Recently uploaded (20)

Vietnam AWS Community Day 2018