How Enterprises Like The Weather
Company Leverage the AWS Cloud

Jinesh Varia
Technology Evangelist, AWS
@jinman

Bryson K...
7 Years Young
Amazon S3 launched: March 14th 2006
Cloud
Broad & Deep Services to Support Virtually Any Cloud Workload

Compute
Amazon EC2
Amazon EMR
Amazon ELB

Storage &
CDN

Ne...
Trusted by Enterprises Around the World
Used by Government Agencies & Educational Institutions Worldwide
• $5.2B retail business

Every day, AWS adds enough

• 7,800 employees

server capacity to power that

• A whole lot of se...
Gartner Magic Quadrant for Cloud Infrastructure as a Service
(August 19, 2013)

“AWS is the overwhelming market share
lead...
Bryson Koehler
EVP and CIO
9
B2B2C
DIRECT
CONSUMER
ONLINE & APPS

B2B

TV

GOV’T
PRODUCTS

WHERE THE WORLD GETS ITS WEATHER

61.6M
#1 MOST DISTRIBUTED
...
Did You Know? TWC Weather Services for Businesses
World’s Leading Weather Media Company
 - Power 500+ broadcast channels ...
Weather is Changing
Warming winters in US

•
•
•
•

2012 was the hottest in recorded history for the
US
Last time January ...
Weather changes…
So must IT
Huge projects required to support these
business needs

In the next 12 months TWC will:
• Incr...
Enterprise Strategies
For the Cloud

14
Strategy

Build All New Apps
and Services in the
Cloud

15
Strategy 1: Build New Apps in the Cloud

Faster to build
Social
Apps

Mobile
Streaming

Consumer
Apps

Easier to manage

L...
All of TWC’s new applications are being purposebuilt to run exclusively in the cloud
MAX 3D Radar
PilotBrief

New Weather....
Speed and Agility: Rapidly developing, testing, deploying
in the Cloud

From 1 highly
constrained team
1 release a month

...
Strategy 1: Nasdaq used AWS to Build a New Line of Business
Strategy

Augment On-Premises
IT Resources with Cloud
Capacity

20
Customer
Corporate
Data Center

Availability Zone A
Customer
Gateway

Private
Dedicated 10G
DirectConnect

VPN
Gateway

NA...
Strategy #2
Augment On-Premises
IT Resources with
Cloud Capacity

Use Cloud to Make
On-Premises Apps Better

22
Strategy 2: Use Cloud to Make On-Premises Apps Better

App 1
Analytics

App 2
App 3

…
Corporate
Data Centers

App N

Elas...
Strategy 2: S&P Capital IQ Uses AWS for Big Data Processing

S3

Provides data to
4200+ top global
investment firms

Hadoo...
Weather’s Big Data Analytics Platform for Enterprises and Government
TWC’s Redshift Cluster now
contains:

WeatherFX
Creat...
Weather is more severe

26
Weather ⌃ is more accurate

Day 1-3 Forecast – Temperature and Precipitation
Weather forecast is more accurate
3, 5, 7 Day Forecast 1980-2010
100
90

Accuracy (%)

80
70
3 day

60

5 day
7 day

50
40...
Taking the World’s most accurate weather forecast into the future and moving
it to the Cloud…
Elastic Load Balancing

Data...
Lots of
Users

Elastic Load Balancing

Data

Data API
Caching, Notifications, Auth etc.

Government
Autoscaling
Instances
...
Lots of
Users

Weather.com +
Weather Channel, etc

Data
Government
Autoscaling
Instances
US-EAST

Data
Universities
Resear...
Lots of
Users
SEVERE WEATHER

Scaling to provide the
best accuracy

Data

Improved frequency of forecast
updates, more loc...
Strategy #2
Augment On-Premises
IT Resources with
Cloud Capacity

Use Cloud to Make
On-Premises Apps Better

33
Strategy #2
Augment On-Premises
IT Resources with
Cloud Capacity

Use Cloud to Make
On-Premises Apps Better

Build Cloud A...
Strategy 2: Cloud Apps that Integrate with On-Premises Apps
AWS serves
application
content & data
Integration to
Samsung d...
Hybrid App: TWC Forecast Platform Integrates with Nationally
Distributed Cable Network
Legacy Star Distribution
August 201...
Strategy

Migrate Existing
Apps & Data to the
Cloud

37
Strategy 3: Migrate Existing Apps and Data to the Cloud

1/3 of servers
migrated to AWS

App

On-Premises
Data Centers

Mi...
Our First Real AWS Project : Migrate real-time radar mapping system
wunderground.com to AWS Cloud

Wunderground Radar and ...
30,000
Personal
Weather
Stations
Source: Wunderground, Inc. 2013
Before Migration – Traditional IT Model doesn’t scale well

Server Count

Avg. CPU Load

(110 Servers)

HTTP Response Late...
Strategy
All-in Cloud

42
Strategy 4: All-In
100s of applications
supporting 33M+
global members

10,000s of EC2
instances in multiple
regions & zon...
is All-in
TWC’s transformation into a large scale, global, big data platform for the
world’s weather is founded on a All-I...
Strategy 4: TWC’s All-In Cloud Strategy

We are using all 3 strategies
simultaneously across the
enterprise to get us ther...
Key Takeaway: Cloud isn’t an ‘All or Nothing’ Choice

Strategy #1

Strategy #2

Strategy #3

Strategy #4

Build All New
Ap...
Thank You!
Jinesh Varia
Technology Evangelist, AWS
jvaria@amazon.com
@jinman

Bryson Koehler
EVP, CIO, The Weather Company...
Upcoming SlideShare
Loading in...5
×

How Enterprises like The Weather Company use the AWS Cloud - Jinesh Varia & Bryson Koelher

1,603

Published on

TIA Keynote 2013: Jinesh Varia, Technology Evangelist, AWS and Bryson Koehler, Executive Vice President and CIO, The Weather Company discuss how different enterprises use AWS today. They discuss 4 strategies and real customer stories that represent trends in how large-scale enterprises adopt cloud computing. For the first time, The Weather Company, discusses on stage their strategy and use of AWS.

Video available here: www.youtube.com/watch?v=crIxor6FqCg

Published in: Technology, Business
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
1,603
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
0
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

How Enterprises like The Weather Company use the AWS Cloud - Jinesh Varia & Bryson Koelher

  1. 1. How Enterprises Like The Weather Company Leverage the AWS Cloud Jinesh Varia Technology Evangelist, AWS @jinman Bryson Koehler EVP, CIO, The Weather Company @brysonkoehler 1
  2. 2. 7 Years Young Amazon S3 launched: March 14th 2006
  3. 3. Cloud
  4. 4. Broad & Deep Services to Support Virtually Any Cloud Workload Compute Amazon EC2 Amazon EMR Amazon ELB Storage & CDN Networking Amazon Route 53 Amazon VPC AWS Direct Connect AWS Partner Network Amazon S3 Amazon Glacier Amazon EBS AWS Import/Exp AWS Premium Support Database Amazon RDS Amazon DynamoDB Amazon Elasticache Amazon RedShift AWS Professional Services App Services Management CloudSearch Amazon SWF Amazon SQS Amazon SNS Amazon SES Amazon Elastic Transcoder AWS IAM Amazon CloudWatch AWS Elastic Beanstalk AWS Cloudformation AWS Data Pipeline AWS OpsWorks AWS CloudHSM AWS Trusted Advisor AWS Marketplace AWS Training & Certification
  5. 5. Trusted by Enterprises Around the World
  6. 6. Used by Government Agencies & Educational Institutions Worldwide
  7. 7. • $5.2B retail business Every day, AWS adds enough • 7,800 employees server capacity to power that • A whole lot of servers whole $5B enterprise
  8. 8. Gartner Magic Quadrant for Cloud Infrastructure as a Service (August 19, 2013) “AWS is the overwhelming market share leader, with more than five times the compute capacity in use than the aggregate total of the other fourteen providers in this Magic Quadrant.” Gartner “Magic Quadrant for Cloud Infrastructure as a Service,” Lydia Leong, Douglas Toombs, Bob Gill, Gregor Petri, Tiny Haynes, August 19, 2013. This Magic Quadrant graphic was published by Gartner, Inc. as part of a larger research note and should be evaluated in the context of the entire report.. The Gartner report is available upon request from Steven Armstrong (asteven@amazon.com). Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings. Gartner research publications consist of the opinions of Gartner's research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.
  9. 9. Bryson Koehler EVP and CIO 9
  10. 10. B2B2C DIRECT CONSUMER ONLINE & APPS B2B TV GOV’T PRODUCTS WHERE THE WORLD GETS ITS WEATHER 61.6M #1 MOST DISTRIBUTED Cable Network* *Nielsen, October 2012. Unduplicated Monthly Uniques (weather.com + wunderground)* 72% visit 2x or more Daily** *comScore, Oct 2012 **Omniture, July 2012 100M+ App Downloads* #2 All Time Most Downloaded iPad App** 47.2M Unduplicated Monthly Uniques*** *Various app stores, TWC only, **App Store Rewind, 2011 ***Omniture Site Catalyst, August 2012. TWC Mobile & wunderground 10
  11. 11. Did You Know? TWC Weather Services for Businesses World’s Leading Weather Media Company  - Power 500+ broadcast channels world-wide World’s Leading Aviation Weather Service  - Support 50,000+ flights daily with embedded forecasters, systems and tools  - 85% of major US airlines. 30% of Top 100 global. World’s Leading Energy Forecaster  - Over 300 power trading clients worldwide  - Hourly, daily, and seasonal trading Fastest Growing Insurance Weather Service  - 7 of the top 10 P&C insurance customers  - Connecting Marketing, Underwriting, & Claims with systems & tools 11
  12. 12. Weather is Changing Warming winters in US • • • • 2012 was the hottest in recorded history for the US Last time January was colder than normal was 1983 In July 2012 61% of the country was in drought conditions 9.2MM acres charred in 2012; 3rd highest in 13 years Source: Time.com; Climate Central
  13. 13. Weather changes… So must IT Huge projects required to support these business needs In the next 12 months TWC will: • Increase data points by 1,400% • Increase data velocity by 300% • Increase number of deployments by 2,000% • Increase number of TV feeds from 1 to 7 • Increase the amount of video content by 100% • Increase the amount of social content by 500% ... Cloud-Ready • Consolidate 13 Data Centers to 4 • 75% of workloads to IaaS Cloud • Move to agile development teams • New TV Transmission System (retire 2, build 1) • 575 APIs and 20 Data platforms to 1 platform • 2500+ new localization servers • New HR, Email, Finance, Inventory, Expense, etc.
  14. 14. Enterprise Strategies For the Cloud 14
  15. 15. Strategy Build All New Apps and Services in the Cloud 15
  16. 16. Strategy 1: Build New Apps in the Cloud Faster to build Social Apps Mobile Streaming Consumer Apps Easier to manage Less expensive to run Distributed architectures for high availability Product Websites Global Web Sites Marketing Campaigns
  17. 17. All of TWC’s new applications are being purposebuilt to run exclusively in the cloud MAX 3D Radar PilotBrief New Weather.com and Other Websites New Weather Analytics Systems New Aviation/Energy Forecasting Apps New Data Ingest & Publishing Apps Weather Data API Mobile Apps 17
  18. 18. Speed and Agility: Rapidly developing, testing, deploying in the Cloud From 1 highly constrained team 1 release a month Core Development Team Extra Development Resources 8-10 decoupled Scrum teams 4 releases a day Contractor Team Remote Team
  19. 19. Strategy 1: Nasdaq used AWS to Build a New Line of Business
  20. 20. Strategy Augment On-Premises IT Resources with Cloud Capacity 20
  21. 21. Customer Corporate Data Center Availability Zone A Customer Gateway Private Dedicated 10G DirectConnect VPN Gateway NAT Private Subnet Router Customer Branch Offices Internet Public Subnet Internet Gateway Amazon VPC Availability Zone B The Cloud-Ready Enterprise IT Network Architecture Object Store S3 NoSQL DB Service Email Service Queue Service DynamoDB SES SQS AWS Region
  22. 22. Strategy #2 Augment On-Premises IT Resources with Cloud Capacity Use Cloud to Make On-Premises Apps Better 22
  23. 23. Strategy 2: Use Cloud to Make On-Premises Apps Better App 1 Analytics App 2 App 3 … Corporate Data Centers App N Elastic Map Reduce Amazon Redshift AWS Storage Gateway Backup Amazon S3
  24. 24. Strategy 2: S&P Capital IQ Uses AWS for Big Data Processing S3 Provides data to 4200+ top global investment firms Hadoop Cluster Launched Hadoop faster, Learned Hadoop faster
  25. 25. Weather’s Big Data Analytics Platform for Enterprises and Government TWC’s Redshift Cluster now contains: WeatherFX CreativeFX Partner EDW Platform Amazon S3 AudienceFX … App N TWC Private Data Center EDW Amazon Amazon Redshift Redshift 600+ million records of Weather forecast data since March 2005 700+ million records of Current conditions observation data since April 2004 11+ billion records of Int. Civil Aviation Org. (ICAO) Observation data since 1970 TBs of Storm event data from from 1950 - 2012 Weather sentiment survey data ~1 Billion records Bulk loaded in 36 minutes!
  26. 26. Weather is more severe 26
  27. 27. Weather ⌃ is more accurate Day 1-3 Forecast – Temperature and Precipitation
  28. 28. Weather forecast is more accurate 3, 5, 7 Day Forecast 1980-2010 100 90 Accuracy (%) 80 70 3 day 60 5 day 7 day 50 40 30 20 1980 1990 2000 2010 Source: NCEP 28
  29. 29. Taking the World’s most accurate weather forecast into the future and moving it to the Cloud… Elastic Load Balancing Data Data API Caching, Notifications, Auth etc. Government Autoscaling Instances US-EAST Data Universities Research Facilities Data Private Suppliers or Corporates Multiple Sources of Data Weather.com + Weather Channel, etc Autoscaling Instances APAC Forecast Core Proprietary Forecasting Models Daily roll-ups, worded forecasts, indices, etc. in 49 languages Where the science happens Forecast at 10X points worldwide Weather Company’s Next Generation Forecast Platform Media Retail Agriculture Insurance Aviation Energy Trading Ground Transportation Marine Utilities Emergency Response Construction Outdoor Recreation Military Construction Travel & Hospitality B2B Customers 29
  30. 30. Lots of Users Elastic Load Balancing Data Data API Caching, Notifications, Auth etc. Government Autoscaling Instances US-EAST Data Universities Research Facilities Data Private Suppliers or Corporates Multiple Sources of Data Weather.com + Weather Channel, etc Autoscaling Instances APAC Forecast Core Proprietary Forecasting Models Daily roll-ups, worded forecasts, indices, etc. in 49 languages Where the science happens Forecast at 10X points worldwide Weather Company’s Next Generation Forecast Platform Media Retail Agriculture Insurance Aviation Energy Trading Ground Transportation Marine Utilities Emergency Response Construction Outdoor Recreation Military Construction Travel & Hospitality B2B Customers 30
  31. 31. Lots of Users Weather.com + Weather Channel, etc Data Government Autoscaling Instances US-EAST Data Universities Research Facilities Data Autoscaling Instances APAC Scaling based on demand Elastic scaling when huge surge of user requests Private Suppliers or Corporates Multiple Sources of Data Weather Company’s Next Generation Forecast Platform Media Retail Agriculture Insurance Aviation Energy Trading Ground Transportation Marine Utilities Emergency Response Construction Outdoor Recreation Military Construction Travel & Hospitality B2B Customers 31
  32. 32. Lots of Users SEVERE WEATHER Scaling to provide the best accuracy Data Improved frequency of forecast updates, more locations, less latency Government Autoscaling Instances US-EAST Data Universities Research Facilities Data Autoscaling Instances APAC More Data Sources Private Suppliers or Corporates Multiple Sources of Data sophisticated weighting scheme Weather Company’s Next Generation Forecast Platform Weather.com + Weather Channel, etc. Media Retail Agriculture Insurance Aviation Energy Trading Ground Transportation Marine Utilities Emergency Response Construction Outdoor Recreation Military Construction Travel & Hospitality B2B Customers 32
  33. 33. Strategy #2 Augment On-Premises IT Resources with Cloud Capacity Use Cloud to Make On-Premises Apps Better 33
  34. 34. Strategy #2 Augment On-Premises IT Resources with Cloud Capacity Use Cloud to Make On-Premises Apps Better Build Cloud Apps that Integrate with On-Premises Apps 34
  35. 35. Strategy 2: Cloud Apps that Integrate with On-Premises Apps AWS serves application content & data Integration to Samsung data centers for Authentication and Financial Transactions Hybrid App On-Premises Data Centers Saved $34 Million 85% savings versus traditional hosting
  36. 36. Hybrid App: TWC Forecast Platform Integrates with Nationally Distributed Cable Network Legacy Star Distribution August 2012 Weather Forecast Platform Hybrid App integrates back to STARs TV Network National Coverage Star Legacy Area Served - August 2012 4000 Junior XL On-Premises Data Centers Local on The 8s 4000+ 36 STAR Boxes in US
  37. 37. Strategy Migrate Existing Apps & Data to the Cloud 37
  38. 38. Strategy 3: Migrate Existing Apps and Data to the Cloud 1/3 of servers migrated to AWS App On-Premises Data Centers Migrated SAP and SharePoint Workloads Saved £1.5 Million Saved $1M over 3 years 50% of Traditional hosting costs Migrated clinical trials simulations platform Simulations in 1.2hrs vs. 60hrs 64% reduction in costs
  39. 39. Our First Real AWS Project : Migrate real-time radar mapping system wunderground.com to AWS Cloud Wunderground Radar and Maps 100 million hits a day One Billion data points per day
  40. 40. 30,000 Personal Weather Stations Source: Wunderground, Inc. 2013
  41. 41. Before Migration – Traditional IT Model doesn’t scale well Server Count Avg. CPU Load (110 Servers) HTTP Response Latency (~6000 ms) After Migration - Wunderground Radar App Server Count (from 110 to 170 Instances) Avg. CPU Load HTTP Response Latency (5-15ms)
  42. 42. Strategy All-in Cloud 42
  43. 43. Strategy 4: All-In 100s of applications supporting 33M+ global members 10,000s of EC2 instances in multiple regions & zones At peak consumes 1/3 of US Internet bandwidth
  44. 44. is All-in TWC’s transformation into a large scale, global, big data platform for the world’s weather is founded on a All-In IaaS strategy.
  45. 45. Strategy 4: TWC’s All-In Cloud Strategy We are using all 3 strategies simultaneously across the enterprise to get us there lightning fast! Strategy #1 Strategy #2 Strategy #3 What we are gaining with the cloud:  Transfer TB’s of weather data every day globally  Manage TB’s of digital content globally  Handle 1 TB/sec at peak  Handle 1 Billion requests per pay  Increase production changes by 20X  Decrease hosting costs by 30%+ 45
  46. 46. Key Takeaway: Cloud isn’t an ‘All or Nothing’ Choice Strategy #1 Strategy #2 Strategy #3 Strategy #4 Build All New Apps for the Cloud Augment On-Premises IT resources with Cloud Capacity Migrate Existing Apps and Data to the Cloud All-in
  47. 47. Thank You! Jinesh Varia Technology Evangelist, AWS jvaria@amazon.com @jinman Bryson Koehler EVP, CIO, The Weather Company bryson@weather.com @brysonkoehler 47

×