As GE Transportation moved their applications to the cloud, they faced operational challenges with monitoring their applications and platforms for availability, performance, and compliance. To address their exact requirements, GE developed a cost effective, scalable monitoring and alerting solution based on fully-managed AWS services. In this session, GE Transportation reviews this capability and discusses its process for implementing the solution. Attendees also learn reusable design patterns using AWS managed services that can be applied in a self-service model to scale efficiently across the enterprise.

1. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
AWS re:INVENT
Operational Monitoring and Alerting
At Scale in GE Transportation
B a l a j i I y e r , S e n i o r C o n s u l t a n t , A m a z o n W e b S e r v i c e s
E r i c M c K e e , I n f r a s t r u c t u r e A r c h i t e c t , G e n e r a l E l e c t r i c
N o v e m b e r 2 9 , 2 0 1 7
E N T 3 4 0

2. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
What to Expect from the Session
• Enterprise Monitoring Landscape
• Enterprise operational challenges
• Monitoring Hybrid cloud scenarios
• Understanding Monitoring in the Cloud
• GE Digital Transformation Journey
• Monitoring and Alerting Solution
• Reusable Design Patterns
• Review learnings and future state

3. Challenges with Enterprise Monitoring
Traditional IT toolset
not built for cloud
scale infrastructure
Maintaining
enterprise-wide
visibility is
challenging
Deploying multiple
products is a
significant overhead
Licensing costs &
complexity
Managing cloud and hybrid environments using a
traditional toolset can be complex and costly
IT infrastructure is increasingly spread across on-premises and in the private
and public cloud

4. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Understanding Cloud Monitoring
Traditional Approach
• Manual configuration of physical and
logical resources
• Spans multiple isolated systems and
processes
• Low to moderate visibility
• Low degree of automation
vs.
• Cloud monitoring is different
• Resources are elastic and ephemeral
• Infrastructure can be tightly coupled
to events
• Cost is a concrete metric that can be
tracked
• All changes and access can be audited

5. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Key Tenets of a Monitoring Solution
Manage hybrid
Architecture
Cross-platform
(Windows/Linux)
Scalable and
auditable
Improve security
and compliance
Easily automate
repetitive tasks
Reduce TCO

6. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
GE Digital Transformation Journey

7. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
GE Digital Transformation Journey
“AWS’s industry leading cloud services have allowed us to push the boundaries, think big, and deliver
better outcomes for GE.” – Chris Drumgoole, CTO,
GE Digital
Platform Services
Monitoring & Alerting
Compliance
Cloud-Based Platform
40% Applications in 2017
Operational Efficiency
Train Performance &
Optimization
Real-time Insights
INNOVATIONPRODUCTIVITY

8. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
GE Transportation — Adoption Landscape
40% Apps
in Cloud
30-50%
Cost out
100% Build
Automation
20% apps
migrate a year
Decrease Downtime
80% Better
Cost
Tracking
< 8 weeks per migration
Rolling
Upgrades
Improved
Availability
Increase in
Performance

9. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Operations and Scaling Challenges
G E T R A N S P O R T A T I O N

10. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Legacy Silos
Compute Storage Network Services

11. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Legacy Silos
Compute Storage Network Services
Tools
Monitors
Alerts
Compliance
Security
Tools
Monitors
Alerts
Compliance
Security
Tools
Monitors
Alerts
Compliance
Security
Tools
Monitors
Alerts
Compliance
Security

12. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Cloud Team Inherits it All
Compute Storage Network Services

13. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Cloud Team Inherits it All
Compute Storage Network
Legacy
Tools
Services
Distant OwnershipIntegration
?

14. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Monitoring & Alerting Challenges
Infrastructure & Agents
Open Source
Commercial
Costs Capabilities Supportability
Entry Cost / High
Management
Monitoring
Alerting
Monitors/Alerts for
Critical Services
Application Support
Platform Maintenance
Compliance Controls
Limited Resources
for Cloud
Outages and
Limited
Instrumentation
DevOps Team
Support
Infrastructure
Resource Changes
Services

15. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Identify Gaps
AWS Cloud Monitoring & Alerting
Security & IT Risk
(CTO Risk)
Platform
(CTO Cloud DevOps)
Application
(App Support Teams)
Compliance
NTPSynchronizing
InfrastructureServices Infrastructure
InternalDNSAvailability
InternetGatewayAvailability
CustomerGatewayAvailability
CustomerGatewayChange
NACLRuleChange
VPCDHCPOptions
SubnetIPThreshold
NetworkReachbacktoGE(443)
NetworkReachbacktoGE(80)
RouteTableChange
BastionServerAvailility(22)
NewRelicAgentInstalled
PublicDNSAvailability
OIDCServiceAvailability
SMTPGatewayAvailability
NATServiceAvailability
NATServerForwardingRequests
NATServiceAvailability
LocalUserAccounts
SSH/RDPRestrictedtoBastion
AMIImage<30DaysOld
QualysAgentInstalled&Running
Anti-VirusAgentRunning
ChefClientCheck-In<180Mins
ToolsSecurityGroupApplied
InstanceCPULow/High
InstanceMemLow/High
InstanceStorageLow
ELBAccessibleIncl.i3
RDSHealth(I/O,Conns)
InstanceCPULoadAvg

16. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Maintain a Bias for Action
G E T R A N S P O R T A T I O N

17. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Rapid Solutioning
• Culture
• Empower
• Informed
Move Fast Fresh Approach Action
• No preconceptions
• Simple requirement
• Open
• Quick small risks
• Success criteria
• Take winnings

18. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Design Focus
Reusable
Monitor to Monitor
Existing code
Share with businesses
Serverless
Reliability
Cost
Management
Scalable
Regular growth
Handle spikes / dips
Avoid planning
Low cost
Zero cost to start
Linear cost growth
No barriers enter/exit

19. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Monitoring and Alerting at Scale
Solution
G E T R A N S P O R T A T I O N

20. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Manage Configuration Drift
• Control configuration details such as
anti-virus settings, iptables, etc.
• Compare actual deployments
against specified configuration
policy, and audit every changes with
change tracking.
• Automatically re-apply policies if
state drift is detected
• OS changes
• Local users and permissions
AWS Cloud
corporate data
center
Systems
Manager
Document
Role-based Access
Control

21. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Automate using extensible frameworks
• Generic framework to express your
workflow as automation steps
• Automate golden image creation
• Fix unreachable EC2 instances
• Reset forgotten passwords
• Create custom workflows
Automation
Document
Run the automation
Role and permissioninput

22. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Serverless Monitoring
SERVICES (ANYTHING)
Changes in
data state
Requests to
endpoints
Changes in
resource state
EVENT SOURCE FUNCTION
Node.js
Python
Java
C#

23. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Run Command
State Manager
InventoryMaintenance Window
Patch Manager Automation Parameter Store
Documents
GitHub
Amazon EC2 Systems Manager
Amazon
S3

24. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Reusable Design Patterns
G E T R A N S P O R T A T I O N

25. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Pattern: Systems Manager and Lambda
EC2
System
Manager
Amazon
S3
logs
AWS
Lambda
Amazon
SNS
messages
IAM
AWS API
roles
AWS
Lambda
(remediation)
DynamoDB

26. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Pattern: Systems Manager and Lambda
EC2
System
Manager
Amazon
S3
logs
AWS
Lambda
Amazon
SNS
messages
IAM
AWS API
roles
AWS
Lambda
(remediation)
DynamoDB
1
1. System
Manager
frequency
30 minutes
Ex. Local
Accounts
2. Lambda
Check
Results,
30 minutes
3. Email,
persistent
logging,
remediation
2
3
Value: Instance Level Agent

27. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Pattern: Lambda
AWS
Lambda
Amazon
SNS
messages
IAM
role
Amazon
S3
logs
AWS
Lambda
(remediation)
DynamoDB

28. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Pattern: Lambda
AWS
Lambda
Amazon
SNS
messages
IAM
role
Amazon
S3
logs
AWS
Lambda
(remediation)
DynamoDB
1
2
1. Lambda
Periodic
Function
Ex.
Security
Group
Rules
2. Email,
persistent
logging,
remediation
Value: Infrastructure Monitors

29. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Pattern: SNS Message Logging
AWS
Lambda
Amazon
SNS
DB Put
Message
(Trigger)
IAM
role
S3
logs
API Call for
Subscribers
DynamoDB

30. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Pattern: SNS Message Logging
AWS
Lambda
Amazon
SNS
DB Put
Message
(Trigger)
IAM
role
S3
logs
API Call for
Subscribers
DynamoDB
1 2
1. Message
Distributed to
email via SNS
2. SNS Topic
Subscriptions,
Message body,
time sent to NoSQL
DynamoDB table
Value: Persistent Communication Record

31. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Pattern: CloudTrail + CloudWatch Rules
Amazon
SNS
Topic
email
AWS
CloudTrail
Amazon
CloudWatch
Alarm

32. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Pattern: CloudTrail + CloudWatch Rules
Amazon
SNS
Topic
email
AWS
CloudTrail
Amazon
CloudWatch
Alarm
1 2
1. Cloud Trail
Record API,
Ex. IAM
Policy
2. Cloudwatch Rule
for specific API
Calls, IAM Policy
Create or Modify
3
3. Notify Risk
Team
Value: Chosen API Call Monitoring

33. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Pattern: AWS Config Rules
AWS
Config
Non-compliant
Resources
AWS
Lambda
Amazon
SNS
AWS
Lambda
(remediation)
Rules

34. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Pattern: AWS Config Rules
AWS
Config
Rules
Non-compliant
Resources
AWS
Lambda
32
1. AWS Config Managed
Rule on Config
Change Rule, Ex. s3
Public Read
2. Periodic Lambda
Function for Non-
compliant
Resources Actions
Amazon
SNS
AWS
Lambda
(remediation)
Value: Critical Resource Changes
1

35. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Benefits of this solution
• Agility
• Scaling components
• Entry cost nearly zero
• Reusable tools
• Operationally simple

36. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Lessons Learned
• Application specific monitors
• SNS unsubscribe
• Timing of reporting > timing of associations
• Spamming risk

37. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
Forward Looking
• Step Functions
• Integrate SNS with a persistent service
• CICD Pipeline for functions
• API Automated Incident management
• Self-healing architectures

38. © 2017, Amazon Web Services, Inc. or its Affiliates. All rights reserved.
THANK YOU!
E r i c M c K e e e r i c . m c k e e @ g e . c o m
B a l a j i I y e r b a l a i y e r @ a m a z o n . c o m
E N T 3 4 0